inconsistent document scores using search_type=dfs_query_then_fetch (how do the _score and _explanation.value fields relate?)


(cole) #1

Hello,

I'm running into issues with inconsistently scored documents between
different index configurations, and I think may be misunderstanding
some of the basics of how scores are calculated. Repro details are
below, but my basic question is how a hit's _score relates to its
_explanation.value (provided when "explain" : true is set in the
query). I understand scores are by default calculated within each
shard, but I thought passing search_type=dfs_query_then_fetch would
lead to the calculation of globally "correct" scores.

Repro steps:

[start with empty index, configured with 1 or 2 shards as called out
below]
curl -XPUT http://localhost:9200/index/mytype/1 -d '{ "name" : "Foo
Bar" }'
curl -XPUT http://localhost:9200/index/mytype/2 -d '{ "name" : "Foo
Zip" }'
curl -XPUT http://localhost:9200/index/mytype/3 -d '{ "name" : "Zap
Foo" }'
curl -XPUT http://localhost:9200/index/mytype/4 -d '{ "name" :
"Something Else" }'

When the index is configure to use one shard, I get the following
results as expected. Note this is the desired result ("Foo Bar" is
scored highest and "Foo Zip" and "Zap Foo" are tied for second best).

curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d '{"query":
{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":true}'
{
"took" : 40,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 1.2290028,
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 1.2290028, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 1.229003,
...
}, {
"_shard" : 0,
...
"_id" : "2",
"_score" : 0.15891947, "_source" : { "name" : "Foo Zip" },
"_explanation" : {
"value" : 0.15891947,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.15891947, "_source" : { "name" : "Zap Foo" },
"_explanation" : {
"value" : 0.15891947,
...
} ]
}
}

When the index is configured to use two shards, I get the following
results as expected. Note these results aren't desirable since "Zap
Foo" is being scored lower than "Foo Zip", but they're expected since
the documents are stored in different shards and
search_type=dfs_query_then_fetch isn't specified.

curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d '{"query":
{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":true}'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 0.72711754,
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 0.7271175,
...
}, {
"_shard" : 1,
...
"_id" : "2",
"_score" : 0.15891947, "_source" : { "name" : "Foo Zip" },
"_explanation" : {
"value" : 0.15891947,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.09494676, "_source" : { "name" : "Zap Foo" },
"_explanation" : {
"value" : 0.09494675,
...
} ]
}
}

In the two previous cases, _explanation.value matches _score for each
hit, which is what I'd expect from the _explanation data structure.
Now, with the 2-shard index again, I'd expect specifying
search_type=dfs_query_then_fetch would provide me with the desired
results of "Foo Bar" matching highest and "Foo Zip" and "Zap Foo"
tying for second best.

curl -XGET 'localhost:9200/index/mytype/_search?
pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":true}'

{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 0.72711754,
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 1.229003,
...
}, {
"_shard" : 1,
...
"_id" : "2",
"_score" : 0.15891947, "_source" : { "name" : "Foo Zip" },
"_explanation" : {
"value" : 0.15891947,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.09494676, "_source" : { "name" : "Zap Foo" },
"_explanation" : {
"value" : 0.15891947,
...
} ]
}
}

Note in this case _explanation.value does not match _score for hits 1
and 3, and _score for hits 2 and 3 differ even though
_explanation.value matches. What is causing _explanation.value to
change for the final _score field? If someone could point out to me
what I'm missing about the relationship between _score and
_explanation.value, I'd very much appreciate it.

Ideally, the results from my search system would be consistent between
indexing runs using generated document IDs (or, stated differently,
the shard allocation would not affect document scoring). Is that
achievable?

Thanks,
Cole


(Shay Banon) #2

I need to double check, but the explanation part will not take the dfs
scoring into account when it shows it (so it will still show shard level
scoring), but the _score will.

On Wed, Dec 14, 2011 at 12:13 AM, cole cole.rottweiler@gmail.com wrote:

Hello,

I'm running into issues with inconsistently scored documents between
different index configurations, and I think may be misunderstanding
some of the basics of how scores are calculated. Repro details are
below, but my basic question is how a hit's _score relates to its
_explanation.value (provided when "explain" : true is set in the
query). I understand scores are by default calculated within each
shard, but I thought passing search_type=dfs_query_then_fetch would
lead to the calculation of globally "correct" scores.

Repro steps:

[start with empty index, configured with 1 or 2 shards as called out
below]
curl -XPUT http://localhost:9200/index/mytype/1 -d '{ "name" : "Foo
Bar" }'
curl -XPUT http://localhost:9200/index/mytype/2 -d '{ "name" : "Foo
Zip" }'
curl -XPUT http://localhost:9200/index/mytype/3 -d '{ "name" : "Zap
Foo" }'
curl -XPUT http://localhost:9200/index/mytype/4 -d '{ "name" :
"Something Else" }'

When the index is configure to use one shard, I get the following
results as expected. Note this is the desired result ("Foo Bar" is
scored highest and "Foo Zip" and "Zap Foo" are tied for second best).

curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d '{"query":
{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":true}'
{
"took" : 40,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 1.2290028,
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 1.2290028, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 1.229003,
...
}, {
"_shard" : 0,
...
"_id" : "2",
"_score" : 0.15891947, "_source" : { "name" : "Foo Zip" },
"_explanation" : {
"value" : 0.15891947,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.15891947, "_source" : { "name" : "Zap Foo" },
"_explanation" : {
"value" : 0.15891947,
...
} ]
}
}

When the index is configured to use two shards, I get the following
results as expected. Note these results aren't desirable since "Zap
Foo" is being scored lower than "Foo Zip", but they're expected since
the documents are stored in different shards and
search_type=dfs_query_then_fetch isn't specified.

curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d '{"query":
{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":true}'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 0.72711754,
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 0.7271175,
...
}, {
"_shard" : 1,
...
"_id" : "2",
"_score" : 0.15891947, "_source" : { "name" : "Foo Zip" },
"_explanation" : {
"value" : 0.15891947,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.09494676, "_source" : { "name" : "Zap Foo" },
"_explanation" : {
"value" : 0.09494675,
...
} ]
}
}

In the two previous cases, _explanation.value matches _score for each
hit, which is what I'd expect from the _explanation data structure.
Now, with the 2-shard index again, I'd expect specifying
search_type=dfs_query_then_fetch would provide me with the desired
results of "Foo Bar" matching highest and "Foo Zip" and "Zap Foo"
tying for second best.

curl -XGET 'localhost:9200/index/mytype/_search?
pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":true}'

{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 0.72711754,
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 1.229003,
...
}, {
"_shard" : 1,
...
"_id" : "2",
"_score" : 0.15891947, "_source" : { "name" : "Foo Zip" },
"_explanation" : {
"value" : 0.15891947,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.09494676, "_source" : { "name" : "Zap Foo" },
"_explanation" : {
"value" : 0.15891947,
...
} ]
}
}

Note in this case _explanation.value does not match _score for hits 1
and 3, and _score for hits 2 and 3 differ even though
_explanation.value matches. What is causing _explanation.value to
change for the final _score field? If someone could point out to me
what I'm missing about the relationship between _score and
_explanation.value, I'd very much appreciate it.

Ideally, the results from my search system would be consistent between
indexing runs using generated document IDs (or, stated differently,
the shard allocation would not affect document scoring). Is that
achievable?

Thanks,
Cole


(cole) #3

If _explanation.value was still showing shard-level scoring, wouldn't
the _explanation.value fields be the same between the second and third
example I gave (the 2-shard example without search_type specified vs.
the 2-shard example with search_type=dfs_query_then_fetch)? In the
second example, _explanation.value is different for hits 2 and 3
(which is what I was attempting to "fix"). In the third example,
_explanation.value is the same for hits 2 and 3, which leads me to
believe the dfs scoring is being reflected in _explanation.value.

-cole

On Dec 13, 2:21 pm, Shay Banon kim...@gmail.com wrote:

I need to double check, but the explanation part will not take the dfs
scoring into account when it shows it (so it will still show shard level
scoring), but the _score will.

On Wed, Dec 14, 2011 at 12:13 AM, cole cole.rottwei...@gmail.com wrote:

Hello,

I'm running into issues with inconsistently scored documents between
different index configurations, and I think may be misunderstanding
some of the basics of how scores are calculated. Repro details are
below, but my basic question is how a hit's _score relates to its
_explanation.value (provided when "explain" : true is set in the
query). I understand scores are by default calculated within each
shard, but I thought passing search_type=dfs_query_then_fetch would
lead to the calculation of globally "correct" scores.

Repro steps:

[start with empty index, configured with 1 or 2 shards as called out
below]
curl -XPUThttp://localhost:9200/index/mytype/1-d '{ "name" : "Foo
Bar" }'
curl -XPUThttp://localhost:9200/index/mytype/2-d '{ "name" : "Foo
Zip" }'
curl -XPUThttp://localhost:9200/index/mytype/3-d '{ "name" : "Zap
Foo" }'
curl -XPUThttp://localhost:9200/index/mytype/4-d '{ "name" :
"Something Else" }'

When the index is configure to use one shard, I get the following
results as expected. Note this is the desired result ("Foo Bar" is
scored highest and "Foo Zip" and "Zap Foo" are tied for second best).

curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d '{"query":
{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":true}'
{
"took" : 40,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 1.2290028,
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 1.2290028, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 1.229003,
...
}, {
"_shard" : 0,
...
"_id" : "2",
"_score" : 0.15891947, "_source" : { "name" : "Foo Zip" },
"_explanation" : {
"value" : 0.15891947,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.15891947, "_source" : { "name" : "Zap Foo" },
"_explanation" : {
"value" : 0.15891947,
...
} ]
}
}

When the index is configured to use two shards, I get the following
results as expected. Note these results aren't desirable since "Zap
Foo" is being scored lower than "Foo Zip", but they're expected since
the documents are stored in different shards and
search_type=dfs_query_then_fetch isn't specified.

curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d '{"query":
{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":true}'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 0.72711754,
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 0.7271175,
...
}, {
"_shard" : 1,
...
"_id" : "2",
"_score" : 0.15891947, "_source" : { "name" : "Foo Zip" },
"_explanation" : {
"value" : 0.15891947,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.09494676, "_source" : { "name" : "Zap Foo" },
"_explanation" : {
"value" : 0.09494675,
...
} ]
}
}

In the two previous cases, _explanation.value matches _score for each
hit, which is what I'd expect from the _explanation data structure.
Now, with the 2-shard index again, I'd expect specifying
search_type=dfs_query_then_fetch would provide me with the desired
results of "Foo Bar" matching highest and "Foo Zip" and "Zap Foo"
tying for second best.

curl -XGET 'localhost:9200/index/mytype/_search?
pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":true}'

{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 0.72711754,
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 1.229003,
...
}, {
"_shard" : 1,
...
"_id" : "2",
"_score" : 0.15891947, "_source" : { "name" : "Foo Zip" },
"_explanation" : {
"value" : 0.15891947,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.09494676, "_source" : { "name" : "Zap Foo" },
"_explanation" : {
"value" : 0.15891947,
...
} ]
}
}

Note in this case _explanation.value does not match _score for hits 1
and 3, and _score for hits 2 and 3 differ even though
_explanation.value matches. What is causing _explanation.value to
change for the final _score field? If someone could point out to me
what I'm missing about the relationship between _score and
_explanation.value, I'd very much appreciate it.

Ideally, the results from my search system would be consistent between
indexing runs using generated document IDs (or, stated differently,
the shard allocation would not affect document scoring). Is that
achievable?

Thanks,
Cole


(cole) #4

Hi Shay,

I've made some progress in debugging the issue I'm seeing with
dfs_query_then_fetch (the search_type=dfs_query_then_fetch behavior
appears to have changed between v0.16.4 and v0.17.0 -- see the full
details of my investigation below).

A simpler example than the one I originally gave is the following:

  • Create a fresh index configured to use 2 shards.

  • Add a few simple documents:
    curl -XPUT http://localhost:9200/index/mytype/1 -d '{ "name" : "Foo
    Bar" }'
    curl -XPUT http://localhost:9200/index/mytype/2 -d '{ "name" :
    "Foo" }'
    curl -XPUT http://localhost:9200/index/mytype/3 -d '{ "name" :
    "Foo" }'

  • Query using the default search_type:
    curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
    '{"query":{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" }
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" }
    }, {
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" }
    } ]
    }
    }

  • Query using search_type=dfs_query_then_fetch (note the results are
    exactly the same as the first query):
    curl -XGET 'localhost:9200/index/mytype/_search?
    pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
    {"must":[{"text":{"name":"Foo Bar"}}]}}}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" }
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" }
    }, {
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" }
    } ]
    }
    }

  • Query with "explain":true using the default search_type and note
    _explanation.value matches _score for each result.
    curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
    '{"query":{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}},
    "explain":"true"}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    "_shard" : 0,
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
    "_explanation" : {
    "value" : 0.7271175,
    ...
    }, {
    "_shard" : 0,
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1519148,
    ...
    }, {
    "_shard" : 1,
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.04500804,
    ...
    } ]
    }
    }

  • Query with "explain":true using search_type=def_query_then_fetch and
    note the following:

    • _explanation.value differ from _score
    • _explanation.value differs from _explanation.value in the default
      search_type query
    • _score matches _score from the default search_type query for all
      results

    curl -XGET 'localhost:9200/index/mytype/_search?
    pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
    {"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":"true"}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    "_shard" : 0,
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
    "_explanation" : {
    "value" : 0.98479235,
    ...
    }, {
    "_shard" : 0,
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1610101,
    ...
    }, {
    "_shard" : 1,
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1610101,
    ...
    } ]
    }
    }

It looks like search_type=dfs_query_then_fetch is being taken into
account in the score calculation (expressed in _explanation.value),
but the shard-level score is actually being returned for _score.

Note I've been using v0.18.5 for all of these examples (apologies for
not mentioning that earlier). Using an identical set of repro steps,
I see the same issue in v0.17.0, but I don't see the same issue in
v0.16.4. With v0.16.4, I get the following:

  • Query with "explain":true using the default search_type and note
    _explanation.value matches _score for each result.
    curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
    '{"query":{"bool":{"must":[{"text":{"name":"Foo
    Bar"}}]}},"explain":true}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
    "_explanation" : {
    "value" : 0.7271175,
    ...
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1519148,
    ...
    }, {
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.04500804,
    ...
    } ]
    }
    }

  • Query with "explain":true using search_type=def_query_then_fetch and
    note the following:

    • _explanation.value matches _score
    • _explanation.value (and _score) differs from _explanation.value in
      the default search_type query

    curl -XGET 'localhost:9200/index/mytype/_search?
    pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
    {"must":[{"text":{"name":"Foo Bar"}}]}},"explain":true}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.98479235, "_source" : { "name" : "Foo Bar" },
    "_explanation" : {
    "value" : 0.98479235,
    ...
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1610101, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1610101,
    ...
    }, {
    ...
    "_id" : "2",
    "_score" : 0.1610101, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1610101,
    ...
    } ]
    }
    }

It looks like something changed between v0.16.4 and v0.17.0 that
caused _score to not be set to _explanation.value, and it looks like
_explanation.value is taking the dfs portion of the search into
account. Can I assume the behavior in v0.16.4 is correct? Or was the
scoring behavior intentionally changed in v0.17.0? I'll try to find
the actual code change in the elasticsearch that caused this (I'd
appreciate any pointers you may have since I'm not familiar with the
codebase).

Thanks,
Cole

On Dec 13, 3:01 pm, cole cole.rottwei...@gmail.com wrote:

If _explanation.value was still showing shard-level scoring, wouldn't
the _explanation.value fields be the same between the second and third
example I gave (the 2-shard example without search_type specified vs.
the 2-shard example with search_type=dfs_query_then_fetch)? In the
second example, _explanation.value is different for hits 2 and 3
(which is what I was attempting to "fix"). In the third example,
_explanation.value is the same for hits 2 and 3, which leads me to
believe the dfs scoring is being reflected in _explanation.value.

-cole

On Dec 13, 2:21 pm, Shay Banon kim...@gmail.com wrote:

I need to double check, but the explanation part will not take the dfs
scoring into account when it shows it (so it will still show shard level
scoring), but the _score will.

On Wed, Dec 14, 2011 at 12:13 AM, cole cole.rottwei...@gmail.com wrote:

Hello,

I'm running into issues with inconsistently scored documents between
different index configurations, and I think may be misunderstanding
some of the basics of how scores are calculated. Repro details are
below, but my basic question is how a hit's _score relates to its
_explanation.value (provided when "explain" : true is set in the
query). I understand scores are by default calculated within each
shard, but I thought passing search_type=dfs_query_then_fetch would
lead to the calculation of globally "correct" scores.

Repro steps:

[start with empty index, configured with 1 or 2 shards as called out
below]
curl -XPUThttp://localhost:9200/index/mytype/1-d'{ "name" : "Foo
Bar" }'
curl -XPUThttp://localhost:9200/index/mytype/2-d'{ "name" : "Foo
Zip" }'
curl -XPUThttp://localhost:9200/index/mytype/3-d'{ "name" : "Zap
Foo" }'
curl -XPUThttp://localhost:9200/index/mytype/4-d'{ "name" :
"Something Else" }'

When the index is configure to use one shard, I get the following
results as expected. Note this is the desired result ("Foo Bar" is
scored highest and "Foo Zip" and "Zap Foo" are tied for second best).

curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d '{"query":
{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":true}'
{
"took" : 40,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 1.2290028,
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 1.2290028, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 1.229003,
...
}, {
"_shard" : 0,
...
"_id" : "2",
"_score" : 0.15891947, "_source" : { "name" : "Foo Zip" },
"_explanation" : {
"value" : 0.15891947,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.15891947, "_source" : { "name" : "Zap Foo" },
"_explanation" : {
"value" : 0.15891947,
...
} ]
}
}

When the index is configured to use two shards, I get the following
results as expected. Note these results aren't desirable since "Zap
Foo" is being scored lower than "Foo Zip", but they're expected since
the documents are stored in different shards and
search_type=dfs_query_then_fetch isn't specified.

curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d '{"query":
{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":true}'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 0.72711754,
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 0.7271175,
...
}, {
"_shard" : 1,
...
"_id" : "2",
"_score" : 0.15891947, "_source" : { "name" : "Foo Zip" },
"_explanation" : {
"value" : 0.15891947,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.09494676, "_source" : { "name" : "Zap Foo" },
"_explanation" : {
"value" : 0.09494675,
...
} ]
}
}

In the two previous cases, _explanation.value matches _score for each
hit, which is what I'd expect from the _explanation data structure.
Now, with the 2-shard index again, I'd expect specifying
search_type=dfs_query_then_fetch would provide me with the desired
results of "Foo Bar" matching highest and "Foo Zip" and "Zap Foo"
tying for second best.

curl -XGET 'localhost:9200/index/mytype/_search?
pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":true}'

{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 0.72711754,
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 1.229003,
...
}, {
"_shard" : 1,
...
"_id" : "2",
"_score" : 0.15891947, "_source" : { "name" : "Foo Zip" },
"_explanation" : {
"value" : 0.15891947,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.09494676, "_source" : { "name" : "Zap Foo" },
"_explanation" : {
"value" : 0.15891947,
...
} ]
}
}

Note in this case _explanation.value does not match _score for hits 1
and 3, and _score for hits 2 and 3 differ even though
_explanation.value matches. What is causing _explanation.value to
change for the final _score field? If someone could point out to me
what I'm missing about the relationship between _score and
_explanation.value, I'd very much appreciate it.

Ideally, the results from my search system would be consistent between
indexing runs using generated document IDs (or, stated differently,
the shard allocation would not affect document scoring). Is that
achievable?

Thanks,
Cole


(cole) #5

Correction: I didn't see version 0.16.5. The behavior of
dfs_query_then_fetch is the same between v0.16.4 and v0.16.5. The
change in behavior is between v0.16.5 and v0.17.0.

-cole

On Dec 15, 11:54 am, cole cole.rottwei...@gmail.com wrote:

Hi Shay,

I've made some progress in debugging the issue I'm seeing with
dfs_query_then_fetch (the search_type=dfs_query_then_fetch behavior
appears to have changed between v0.16.4 and v0.17.0 -- see the full
details of my investigation below).

A simpler example than the one I originally gave is the following:

  • Create a fresh index configured to use 2 shards.

  • Add a few simple documents:
    curl -XPUThttp://localhost:9200/index/mytype/1-d '{ "name" : "Foo
    Bar" }'
    curl -XPUThttp://localhost:9200/index/mytype/2-d '{ "name" :
    "Foo" }'
    curl -XPUThttp://localhost:9200/index/mytype/3-d '{ "name" :
    "Foo" }'

  • Query using the default search_type:
    curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
    '{"query":{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" }
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" }
    }, {
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" }
    } ]
    }

}

  • Query using search_type=dfs_query_then_fetch (note the results are
    exactly the same as the first query):
    curl -XGET 'localhost:9200/index/mytype/_search?
    pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
    {"must":[{"text":{"name":"Foo Bar"}}]}}}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" }
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" }
    }, {
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" }
    } ]
    }

}

  • Query with "explain":true using the default search_type and note
    _explanation.value matches _score for each result.
    curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
    '{"query":{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}},
    "explain":"true"}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    "_shard" : 0,
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
    "_explanation" : {
    "value" : 0.7271175,
    ...
    }, {
    "_shard" : 0,
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1519148,
    ...
    }, {
    "_shard" : 1,
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.04500804,
    ...
    } ]
    }

}

  • Query with "explain":true using search_type=def_query_then_fetch and
    note the following:

    • _explanation.value differ from _score
    • _explanation.value differs from _explanation.value in the default
      search_type query
    • _score matches _score from the default search_type query for all
      results

    curl -XGET 'localhost:9200/index/mytype/_search?
    pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
    {"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":"true"}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    "_shard" : 0,
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
    "_explanation" : {
    "value" : 0.98479235,
    ...
    }, {
    "_shard" : 0,
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1610101,
    ...
    }, {
    "_shard" : 1,
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1610101,
    ...
    } ]
    }

}

It looks like search_type=dfs_query_then_fetch is being taken into
account in the score calculation (expressed in _explanation.value),
but the shard-level score is actually being returned for _score.

Note I've been using v0.18.5 for all of these examples (apologies for
not mentioning that earlier). Using an identical set of repro steps,
I see the same issue in v0.17.0, but I don't see the same issue in
v0.16.4. With v0.16.4, I get the following:

  • Query with "explain":true using the default search_type and note
    _explanation.value matches _score for each result.
    curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
    '{"query":{"bool":{"must":[{"text":{"name":"Foo
    Bar"}}]}},"explain":true}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
    "_explanation" : {
    "value" : 0.7271175,
    ...
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1519148,
    ...
    }, {
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.04500804,
    ...
    } ]
    }

}

  • Query with "explain":true using search_type=def_query_then_fetch and
    note the following:

    • _explanation.value matches _score
    • _explanation.value (and _score) differs from _explanation.value in
      the default search_type query

    curl -XGET 'localhost:9200/index/mytype/_search?
    pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
    {"must":[{"text":{"name":"Foo Bar"}}]}},"explain":true}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.98479235, "_source" : { "name" : "Foo Bar" },
    "_explanation" : {
    "value" : 0.98479235,
    ...
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1610101, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1610101,
    ...
    }, {
    ...
    "_id" : "2",
    "_score" : 0.1610101, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1610101,
    ...
    } ]
    }

}

It looks like something changed between v0.16.4 and v0.17.0 that
caused _score to not be set to _explanation.value, and it looks like
_explanation.value is taking the dfs portion of the search into
account. Can I assume the behavior in v0.16.4 is correct? Or was the
scoring behavior intentionally changed in v0.17.0? I'll try to find
the actual code change in the elasticsearch that caused this (I'd
appreciate any pointers you may have since I'm not familiar with the
codebase).

Thanks,
Cole

On Dec 13, 3:01 pm, cole cole.rottwei...@gmail.com wrote:

If _explanation.value was still showing shard-level scoring, wouldn't
the _explanation.value fields be the same between the second and third
example I gave (the 2-shard example without search_type specified vs.
the 2-shard example with search_type=dfs_query_then_fetch)? In the
second example, _explanation.value is different for hits 2 and 3
(which is what I was attempting to "fix"). In the third example,
_explanation.value is the same for hits 2 and 3, which leads me to
believe the dfs scoring is being reflected in _explanation.value.

-cole

On Dec 13, 2:21 pm, Shay Banon kim...@gmail.com wrote:

I need to double check, but the explanation part will not take the dfs
scoring into account when it shows it (so it will still show shard level
scoring), but the _score will.

On Wed, Dec 14, 2011 at 12:13 AM, cole cole.rottwei...@gmail.com wrote:

Hello,

I'm running into issues with inconsistently scored documents between
different index configurations, and I think may be misunderstanding
some of the basics of how scores are calculated. Repro details are
below, but my basic question is how a hit's _score relates to its
_explanation.value (provided when "explain" : true is set in the
query). I understand scores are by default calculated within each
shard, but I thought passing search_type=dfs_query_then_fetch would
lead to the calculation of globally "correct" scores.

Repro steps:

[start with empty index, configured with 1 or 2 shards as called out
below]
curl -XPUThttp://localhost:9200/index/mytype/1-d'{ "name" : "Foo
Bar" }'
curl -XPUThttp://localhost:9200/index/mytype/2-d'{ "name" : "Foo
Zip" }'
curl -XPUThttp://localhost:9200/index/mytype/3-d'{ "name" : "Zap
Foo" }'
curl -XPUThttp://localhost:9200/index/mytype/4-d'{ "name" :
"Something Else" }'

When the index is configure to use one shard, I get the following
results as expected. Note this is the desired result ("Foo Bar" is
scored highest and "Foo Zip" and "Zap Foo" are tied for second best).

curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d '{"query":
{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":true}'
{
"took" : 40,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 1.2290028,
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 1.2290028, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 1.229003,
...
}, {
"_shard" : 0,
...
"_id" : "2",
"_score" : 0.15891947, "_source" : { "name" : "Foo Zip" },
"_explanation" : {
"value" : 0.15891947,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.15891947, "_source" : { "name" : "Zap Foo" },
"_explanation" : {
"value" : 0.15891947,
...
} ]
}
}

When the index is configured to use two shards, I get the following
results as expected. Note these

...

read more »


(cole) #6

Commit 882ccf32c826eed14950ba64c76df656ef45c01e caused the change in
dfs_query_then_fetch behavior. Specifically, this following change is
responsible:

diff --git a/modules/elasticsearch/src/main/java/org/elasticsearch/
search/internal/ContextIndexSearcher.java b/modules/elasticsearch/src/
main/java/org/elasticsearch/search/internal/ContextIndexSearcher.java
index f09683f..9c1149b 100644
--- a/modules/elasticsearch/src/main/java/org/elasticsearch/search/
internal/ContextIndexSearcher.java
+++ b/modules/elasticsearch/src/main/java/org/elasticsearch/search/
internal/ContextIndexSearcher.java
@@ -131,10 +131,11 @@ public class ContextIndexSearcher extends
ExtendedIndexSearcher {
}

 @Override public Weight createNormalizedWeight(Query query)

throws IOException {

  •    if (dfSource == null) {
    
  •        return super.createNormalizedWeight(query);
    
  •    // if its the main query, use we have dfs data, only then do
    

it

  •    if (dfSource != null && (query == searchContext.query() ||
    

query == searchContext.parsedQuery().query())) {

  •        return dfSource.createNormalizedWeight(query);
       }
    
  •    return dfSource.createNormalizedWeight(query);
    
  •    return super.createNormalizedWeight(query);
    
    }

Shay, can you shed some light on that change (the commit message was
"only use dfs data on main query executed")? If I change the code
back on the v0.18.5 tag, I get the dfs_query_then_fetch behavior I
expect, but I'm assuming the code you added affects more complex query
types (parent-child queries by the looks of the unit test added in the
same commit). Did you intend for the change to also affect the basic
dfs_query_then_fetch behavior?

Thanks,
Cole

On Dec 15, 12:28 pm, cole cole.rottwei...@gmail.com wrote:

Correction: I didn't see version 0.16.5. The behavior of
dfs_query_then_fetch is the same between v0.16.4 and v0.16.5. The
change in behavior is between v0.16.5 and v0.17.0.

-cole

On Dec 15, 11:54 am, cole cole.rottwei...@gmail.com wrote:

Hi Shay,

I've made some progress in debugging the issue I'm seeing with
dfs_query_then_fetch (the search_type=dfs_query_then_fetch behavior
appears to have changed between v0.16.4 and v0.17.0 -- see the full
details of my investigation below).

A simpler example than the one I originally gave is the following:

  • Create a fresh index configured to use 2 shards.
  • Add a few simple documents:
    curl -XPUThttp://localhost:9200/index/mytype/1-d'{ "name" : "Foo
    Bar" }'
    curl -XPUThttp://localhost:9200/index/mytype/2-d'{ "name" :
    "Foo" }'
    curl -XPUThttp://localhost:9200/index/mytype/3-d'{ "name" :
    "Foo" }'
  • Query using the default search_type:
    curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
    '{"query":{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" }
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" }
    }, {
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" }
    } ]
    }

}

  • Query using search_type=dfs_query_then_fetch (note the results are
    exactly the same as the first query):
    curl -XGET 'localhost:9200/index/mytype/_search?
    pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
    {"must":[{"text":{"name":"Foo Bar"}}]}}}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" }
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" }
    }, {
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" }
    } ]
    }

}

  • Query with "explain":true using the default search_type and note
    _explanation.value matches _score for each result.
    curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
    '{"query":{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}},
    "explain":"true"}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    "_shard" : 0,
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
    "_explanation" : {
    "value" : 0.7271175,
    ...
    }, {
    "_shard" : 0,
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1519148,
    ...
    }, {
    "_shard" : 1,
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.04500804,
    ...
    } ]
    }

}

  • Query with "explain":true using search_type=def_query_then_fetch and
    note the following:
    • _explanation.value differ from _score
    • _explanation.value differs from _explanation.value in the default
      search_type query
    • _score matches _score from the default search_type query for all
      results

curl -XGET 'localhost:9200/index/mytype/_search?
pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":"true"}'
{
...
"hits" : {
...
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 0.98479235,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.1519148, "_source" : { "name" : "Foo" },
"_explanation" : {
"value" : 0.1610101,
...
}, {
"_shard" : 1,
...
"_id" : "2",
"_score" : 0.04500804, "_source" : { "name" : "Foo" },
"_explanation" : {
"value" : 0.1610101,
...
} ]
}

}

It looks like search_type=dfs_query_then_fetch is being taken into
account in the score calculation (expressed in _explanation.value),
but the shard-level score is actually being returned for _score.

Note I've been using v0.18.5 for all of these examples (apologies for
not mentioning that earlier). Using an identical set of repro steps,
I see the same issue in v0.17.0, but I don't see the same issue in
v0.16.4. With v0.16.4, I get the following:

  • Query with "explain":true using the default search_type and note
    _explanation.value matches _score for each result.
    curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
    '{"query":{"bool":{"must":[{"text":{"name":"Foo
    Bar"}}]}},"explain":true}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
    "_explanation" : {
    "value" : 0.7271175,
    ...
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1519148,
    ...
    }, {
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.04500804,
    ...
    } ]
    }

}

  • Query with "explain":true using search_type=def_query_then_fetch and
    note the following:
    • _explanation.value matches _score
    • _explanation.value (and _score) differs from _explanation.value in
      the default search_type query

curl -XGET 'localhost:9200/index/mytype/_search?
pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
{"must":[{"text":{"name":"Foo Bar"}}]}},"explain":true}'
{
...
"hits" : {
...
"hits" : [ {
...
"_id" : "1",
"_score" : 0.98479235, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 0.98479235,
...
}, {
...
"_id" : "3",
"_score" : 0.1610101, "_source" : { "name" : "Foo" },
"_explanation" : {
"value" : 0.1610101,
...
}, {
...
"_id" : "2",
"_score" : 0.1610101, "_source" : { "name" : "Foo" },
"_explanation" : {
"value" : 0.1610101,
...
} ]
}

}

It looks like something changed between v0.16.4 and v0.17.0 that
caused _score to not be set to _explanation.value, and it looks like
_explanation.value is taking the dfs portion of the search into
account. Can I assume the behavior in v0.16.4 is correct? Or was the
scoring behavior intentionally changed in v0.17.0? I'll try to find
the actual code change in the elasticsearch that caused this (I'd
appreciate any pointers you may have since I'm not familiar with the
codebase).

Thanks,
Cole

On Dec 13, 3:01 pm, cole cole.rottwei...@gmail.com wrote:

If _explanation.value was still showing shard-level scoring, wouldn't
the _explanation.value fields be the same between the second and third
example I gave (the 2-shard example without search_type specified vs.
the 2-shard example with search_type=dfs_query_then_fetch)? In the
second example, _explanation.value is different for hits 2 and 3
(which is what I was attempting to "fix"). In the third example,
_explanation.value is the same for hits 2 and 3, which leads me to
believe the dfs scoring is being reflected in _explanation.value.

-cole

On Dec 13, 2:21 pm, Shay Banon kim...@gmail.com wrote:

I need to double check, but the explanation part will not take the dfs
scoring into account when it shows it (so it will still show shard level
scoring), but the _score will.

On Wed, Dec 14, 2011 at 12:13 AM, cole cole.rottwei...@gmail.com wrote:

Hello,

I'm running into issues with inconsistently scored documents between
different index configurations, and I think may be misunderstanding
some of the basics of how scores are calculated. Repro details are
below, but my basic question is how a hit's _score relates to its
_explanation.value (provided when "explain" : true is set in the
query). I understand scores are by default calculated within each
shard, but I thought passing search_type=dfs_query_then_fetch would
lead to the calculation of globally "correct" scores.

Repro steps:

[start with empty index, configured with 1 or 2 shards as called out
below]
curl -XPUThttp://localhost:9200/index/mytype/1-d'{ "name" : "Foo
Bar" }'
curl -XPUThttp://localhost:9200/index/mytype/2-d'{ "name" : "Foo
Zip" }'
curl -XPUThttp://localhost:9200/index/mytype/3-d'{ "name" : "Zap
Foo" }'
curl -XPUThttp://localhost:9200/index/mytype/4-d'{ "name" :
"Something Else" }'

When the index is configure to use one shard, I get the following
results as expected. Note this is the desired result ("Foo Bar" is
scored highest and "Foo Zip" and "Zap Foo" are tied for second best).

curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d '{"query":
{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":true}'
{

...

read more »


(Shay Banon) #7

Heya,

Great work tracking this down!. I did not fully understand your problem
(next time, its better to have code samples in a gist, and a description in
the mail, otherwise, its really hard to make sense of it).

The problem happen because you search against the type as well. What
happens is that the search will be automatically filtered by the type, and
then it will not match the "provided" query to apply the dfs cals. Opened
an issue: https://github.com/elasticsearch/elasticsearch/issues/1546.

On Fri, Dec 16, 2011 at 2:01 AM, cole cole.rottweiler@gmail.com wrote:

Commit 882ccf32c826eed14950ba64c76df656ef45c01e caused the change in
dfs_query_then_fetch behavior. Specifically, this following change is
responsible:

diff --git a/modules/elasticsearch/src/main/java/org/elasticsearch/
search/internal/ContextIndexSearcher.java b/modules/elasticsearch/src/
main/java/org/elasticsearch/search/internal/ContextIndexSearcher.java
index f09683f..9c1149b 100644
--- a/modules/elasticsearch/src/main/java/org/elasticsearch/search/
internal/ContextIndexSearcher.java
+++ b/modules/elasticsearch/src/main/java/org/elasticsearch/search/
internal/ContextIndexSearcher.java
@@ -131,10 +131,11 @@ public class ContextIndexSearcher extends
ExtendedIndexSearcher {
}

@Override public Weight createNormalizedWeight(Query query)

throws IOException {

  •    if (dfSource == null) {
    
  •        return super.createNormalizedWeight(query);
    
  •    // if its the main query, use we have dfs data, only then do
    

it

  •    if (dfSource != null && (query == searchContext.query() ||
    

query == searchContext.parsedQuery().query())) {

  •        return dfSource.createNormalizedWeight(query);
      }
    
  •    return dfSource.createNormalizedWeight(query);
    
  •    return super.createNormalizedWeight(query);
    
    }

Shay, can you shed some light on that change (the commit message was
"only use dfs data on main query executed")? If I change the code
back on the v0.18.5 tag, I get the dfs_query_then_fetch behavior I
expect, but I'm assuming the code you added affects more complex query
types (parent-child queries by the looks of the unit test added in the
same commit). Did you intend for the change to also affect the basic
dfs_query_then_fetch behavior?

Thanks,
Cole

On Dec 15, 12:28 pm, cole cole.rottwei...@gmail.com wrote:

Correction: I didn't see version 0.16.5. The behavior of
dfs_query_then_fetch is the same between v0.16.4 and v0.16.5. The
change in behavior is between v0.16.5 and v0.17.0.

-cole

On Dec 15, 11:54 am, cole cole.rottwei...@gmail.com wrote:

Hi Shay,

I've made some progress in debugging the issue I'm seeing with
dfs_query_then_fetch (the search_type=dfs_query_then_fetch behavior
appears to have changed between v0.16.4 and v0.17.0 -- see the full
details of my investigation below).

A simpler example than the one I originally gave is the following:

  • Create a fresh index configured to use 2 shards.
  • Add a few simple documents:
    curl -XPUThttp://localhost:9200/index/mytype/1-d'{ "name" : "Foo
    Bar" }'
    curl -XPUThttp://localhost:9200/index/mytype/2-d'{ "name" :
    "Foo" }'
    curl -XPUThttp://localhost:9200/index/mytype/3-d'{ "name" :
    "Foo" }'
  • Query using the default search_type:
    curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
    '{"query":{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" }
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" }
    }, {
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" }
    } ]
    }

}

  • Query using search_type=dfs_query_then_fetch (note the results are
    exactly the same as the first query):
    curl -XGET 'localhost:9200/index/mytype/_search?
    pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
    {"must":[{"text":{"name":"Foo Bar"}}]}}}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" }
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" }
    }, {
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" }
    } ]
    }

}

  • Query with "explain":true using the default search_type and note
    _explanation.value matches _score for each result.
    curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
    '{"query":{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}},
    "explain":"true"}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    "_shard" : 0,
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
    "_explanation" : {
    "value" : 0.7271175,
    ...
    }, {
    "_shard" : 0,
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1519148,
    ...
    }, {
    "_shard" : 1,
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.04500804,
    ...
    } ]
    }

}

  • Query with "explain":true using search_type=def_query_then_fetch and
    note the following:
    • _explanation.value differ from _score
    • _explanation.value differs from _explanation.value in the default
      search_type query
    • _score matches _score from the default search_type query for all
      results

curl -XGET 'localhost:9200/index/mytype/_search?
pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":"true"}'
{
...
"hits" : {
...
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 0.98479235,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.1519148, "_source" : { "name" : "Foo" },
"_explanation" : {
"value" : 0.1610101,
...
}, {
"_shard" : 1,
...
"_id" : "2",
"_score" : 0.04500804, "_source" : { "name" : "Foo" },
"_explanation" : {
"value" : 0.1610101,
...
} ]
}

}

It looks like search_type=dfs_query_then_fetch is being taken into
account in the score calculation (expressed in _explanation.value),
but the shard-level score is actually being returned for _score.

Note I've been using v0.18.5 for all of these examples (apologies for
not mentioning that earlier). Using an identical set of repro steps,
I see the same issue in v0.17.0, but I don't see the same issue in
v0.16.4. With v0.16.4, I get the following:

  • Query with "explain":true using the default search_type and note
    _explanation.value matches _score for each result.
    curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
    '{"query":{"bool":{"must":[{"text":{"name":"Foo
    Bar"}}]}},"explain":true}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
    "_explanation" : {
    "value" : 0.7271175,
    ...
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1519148,
    ...
    }, {
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.04500804,
    ...
    } ]
    }

}

  • Query with "explain":true using search_type=def_query_then_fetch and
    note the following:
    • _explanation.value matches _score
    • _explanation.value (and _score) differs from _explanation.value in
      the default search_type query

curl -XGET 'localhost:9200/index/mytype/_search?
pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
{"must":[{"text":{"name":"Foo Bar"}}]}},"explain":true}'
{
...
"hits" : {
...
"hits" : [ {
...
"_id" : "1",
"_score" : 0.98479235, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 0.98479235,
...
}, {
...
"_id" : "3",
"_score" : 0.1610101, "_source" : { "name" : "Foo" },
"_explanation" : {
"value" : 0.1610101,
...
}, {
...
"_id" : "2",
"_score" : 0.1610101, "_source" : { "name" : "Foo" },
"_explanation" : {
"value" : 0.1610101,
...
} ]
}

}

It looks like something changed between v0.16.4 and v0.17.0 that
caused _score to not be set to _explanation.value, and it looks like
_explanation.value is taking the dfs portion of the search into
account. Can I assume the behavior in v0.16.4 is correct? Or was the
scoring behavior intentionally changed in v0.17.0? I'll try to find
the actual code change in the elasticsearch that caused this (I'd
appreciate any pointers you may have since I'm not familiar with the
codebase).

Thanks,
Cole

On Dec 13, 3:01 pm, cole cole.rottwei...@gmail.com wrote:

If _explanation.value was still showing shard-level scoring, wouldn't
the _explanation.value fields be the same between the second and
third

example I gave (the 2-shard example without search_type specified vs.
the 2-shard example with search_type=dfs_query_then_fetch)? In the
second example, _explanation.value is different for hits 2 and 3
(which is what I was attempting to "fix"). In the third example,
_explanation.value is the same for hits 2 and 3, which leads me to
believe the dfs scoring is being reflected in _explanation.value.

-cole

On Dec 13, 2:21 pm, Shay Banon kim...@gmail.com wrote:

I need to double check, but the explanation part will not take the
dfs

scoring into account when it shows it (so it will still show shard
level

scoring), but the _score will.

On Wed, Dec 14, 2011 at 12:13 AM, cole cole.rottwei...@gmail.com
wrote:

Hello,

I'm running into issues with inconsistently scored documents
between

different index configurations, and I think may be
misunderstanding

some of the basics of how scores are calculated. Repro details
are

below, but my basic question is how a hit's _score relates to its
_explanation.value (provided when "explain" : true is set in the
query). I understand scores are by default calculated within
each

shard, but I thought passing search_type=dfs_query_then_fetch
would

lead to the calculation of globally "correct" scores.

Repro steps:

[start with empty index, configured with 1 or 2 shards as called
out

below]
curl -XPUThttp://localhost:9200/index/mytype/1-d'{ "name" : "Foo
Bar" }'
curl -XPUThttp://localhost:9200/index/mytype/2-d'{ "name" : "Foo
Zip" }'
curl -XPUThttp://localhost:9200/index/mytype/3-d'{ "name" : "Zap
Foo" }'
curl -XPUThttp://localhost:9200/index/mytype/4-d'{ "name" :
"Something Else" }'

When the index is configure to use one shard, I get the following
results as expected. Note this is the desired result ("Foo Bar"
is

scored highest and "Foo Zip" and "Zap Foo" are tied for second
best).

curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
'{"query":

{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":true}'
{

...

read more »


(cole) #8

Fantastic. Thanks for the quick fix, Shay. I verified it using my
basic repro case against a local build off change
cb4c249a659b0ecee292f45c7f00aefd8ca0411e.

Apologies for the long emails. I realized how unreadable those were
when I received them myself, and I've resolved to use gist in the
future. =)

Thanks,
Cole

On Dec 16, 9:02 am, Shay Banon kim...@gmail.com wrote:

Heya,

Great work tracking this down!. I did not fully understand your problem
(next time, its better to have code samples in a gist, and a description in
the mail, otherwise, its really hard to make sense of it).

The problem happen because you search against the type as well. What
happens is that the search will be automatically filtered by the type, and
then it will not match the "provided" query to apply the dfs cals. Opened
an issue:https://github.com/elasticsearch/elasticsearch/issues/1546.

On Fri, Dec 16, 2011 at 2:01 AM, cole cole.rottwei...@gmail.com wrote:

Commit 882ccf32c826eed14950ba64c76df656ef45c01e caused the change in
dfs_query_then_fetch behavior. Specifically, this following change is
responsible:

diff --git a/modules/elasticsearch/src/main/java/org/elasticsearch/
search/internal/ContextIndexSearcher.java b/modules/elasticsearch/src/
main/java/org/elasticsearch/search/internal/ContextIndexSearcher.java
index f09683f..9c1149b 100644
--- a/modules/elasticsearch/src/main/java/org/elasticsearch/search/
internal/ContextIndexSearcher.java
+++ b/modules/elasticsearch/src/main/java/org/elasticsearch/search/
internal/ContextIndexSearcher.java
@@ -131,10 +131,11 @@ public class ContextIndexSearcher extends
ExtendedIndexSearcher {
}

@Override public Weight createNormalizedWeight(Query query)

throws IOException {

  •    if (dfSource == null) {
    
  •        return super.createNormalizedWeight(query);
    
  •    // if its the main query, use we have dfs data, only then do
    

it

  •    if (dfSource != null && (query == searchContext.query() ||
    

query == searchContext.parsedQuery().query())) {

  •        return dfSource.createNormalizedWeight(query);
      }
    
  •    return dfSource.createNormalizedWeight(query);
    
  •    return super.createNormalizedWeight(query);
    
    }

Shay, can you shed some light on that change (the commit message was
"only use dfs data on main query executed")? If I change the code
back on the v0.18.5 tag, I get the dfs_query_then_fetch behavior I
expect, but I'm assuming the code you added affects more complex query
types (parent-child queries by the looks of the unit test added in the
same commit). Did you intend for the change to also affect the basic
dfs_query_then_fetch behavior?

Thanks,
Cole

On Dec 15, 12:28 pm, cole cole.rottwei...@gmail.com wrote:

Correction: I didn't see version 0.16.5. The behavior of
dfs_query_then_fetch is the same between v0.16.4 and v0.16.5. The
change in behavior is between v0.16.5 and v0.17.0.

-cole

On Dec 15, 11:54 am, cole cole.rottwei...@gmail.com wrote:

Hi Shay,

I've made some progress in debugging the issue I'm seeing with
dfs_query_then_fetch (the search_type=dfs_query_then_fetch behavior
appears to have changed between v0.16.4 and v0.17.0 -- see the full
details of my investigation below).

A simpler example than the one I originally gave is the following:

  • Create a fresh index configured to use 2 shards.
  • Add a few simple documents:
    curl -XPUThttp://localhost:9200/index/mytype/1-d'{ "name" : "Foo
    Bar" }'
    curl -XPUThttp://localhost:9200/index/mytype/2-d'{ "name" :
    "Foo" }'
    curl -XPUThttp://localhost:9200/index/mytype/3-d'{ "name" :
    "Foo" }'
  • Query using the default search_type:
    curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
    '{"query":{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}}}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" }
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" }
    }, {
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" }
    } ]
    }

}

  • Query using search_type=dfs_query_then_fetch (note the results are
    exactly the same as the first query):
    curl -XGET 'localhost:9200/index/mytype/_search?
    pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
    {"must":[{"text":{"name":"Foo Bar"}}]}}}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" }
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" }
    }, {
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" }
    } ]
    }

}

  • Query with "explain":true using the default search_type and note
    _explanation.value matches _score for each result.
    curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
    '{"query":{"bool":{"must":[{"text":{"name":"Foo Bar"}}]}},
    "explain":"true"}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    "_shard" : 0,
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
    "_explanation" : {
    "value" : 0.7271175,
    ...
    }, {
    "_shard" : 0,
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1519148,
    ...
    }, {
    "_shard" : 1,
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.04500804,
    ...
    } ]
    }

}

  • Query with "explain":true using search_type=def_query_then_fetch and
    note the following:
    • _explanation.value differ from _score
    • _explanation.value differs from _explanation.value in the default
      search_type query
    • _score matches _score from the default search_type query for all
      results

curl -XGET 'localhost:9200/index/mytype/_search?
pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
{"must":[{"text":{"name":"Foo Bar"}}]}}, "explain":"true"}'
{
...
"hits" : {
...
"hits" : [ {
"_shard" : 0,
...
"_id" : "1",
"_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 0.98479235,
...
}, {
"_shard" : 0,
...
"_id" : "3",
"_score" : 0.1519148, "_source" : { "name" : "Foo" },
"_explanation" : {
"value" : 0.1610101,
...
}, {
"_shard" : 1,
...
"_id" : "2",
"_score" : 0.04500804, "_source" : { "name" : "Foo" },
"_explanation" : {
"value" : 0.1610101,
...
} ]
}

}

It looks like search_type=dfs_query_then_fetch is being taken into
account in the score calculation (expressed in _explanation.value),
but the shard-level score is actually being returned for _score.

Note I've been using v0.18.5 for all of these examples (apologies for
not mentioning that earlier). Using an identical set of repro steps,
I see the same issue in v0.17.0, but I don't see the same issue in
v0.16.4. With v0.16.4, I get the following:

  • Query with "explain":true using the default search_type and note
    _explanation.value matches _score for each result.
    curl -XGET 'localhost:9200/index/mytype/_search?pretty=1' -d
    '{"query":{"bool":{"must":[{"text":{"name":"Foo
    Bar"}}]}},"explain":true}'
    {
    ...
    "hits" : {
    ...
    "hits" : [ {
    ...
    "_id" : "1",
    "_score" : 0.72711754, "_source" : { "name" : "Foo Bar" },
    "_explanation" : {
    "value" : 0.7271175,
    ...
    }, {
    ...
    "_id" : "3",
    "_score" : 0.1519148, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.1519148,
    ...
    }, {
    ...
    "_id" : "2",
    "_score" : 0.04500804, "_source" : { "name" : "Foo" },
    "_explanation" : {
    "value" : 0.04500804,
    ...
    } ]
    }

}

  • Query with "explain":true using search_type=def_query_then_fetch and
    note the following:
    • _explanation.value matches _score
    • _explanation.value (and _score) differs from _explanation.value in
      the default search_type query

curl -XGET 'localhost:9200/index/mytype/_search?
pretty=1&search_type=dfs_query_then_fetch' -d '{"query":{"bool":
{"must":[{"text":{"name":"Foo Bar"}}]}},"explain":true}'
{
...
"hits" : {
...
"hits" : [ {
...
"_id" : "1",
"_score" : 0.98479235, "_source" : { "name" : "Foo Bar" },
"_explanation" : {
"value" : 0.98479235,
...
}, {
...
"_id" : "3",
"_score" : 0.1610101, "_source" : { "name" : "Foo" },
"_explanation" : {
"value" : 0.1610101,
...
}, {
...
"_id" : "2",
"_score" : 0.1610101, "_source" : { "name" : "Foo" },
"_explanation" : {
"value" : 0.1610101,
...
} ]
}

}

It looks like something changed

...

read more »


(system) #9