My es version is 6.8. I make a query like below and the field analyzer is a ngram. In the response the unexpected document score is too high. I try to analyze what happen in the query, then i'm confused.
GET /_search
{
"explain": true,
"size": 100,
"query": {
"bool": {
"must": [
{
"match_phrase_prefix": {
"sku_name": {
"query": "LS",
"slop": 0,
"max_expansions": 50,
"boost": 1.0
}
}
}
],
"filter": [
{
"term": {
"visible": {
"value": true,
"boost": 1.0
}
}
},
{
"term": {
"deleted": {
"value": false,
"boost": 1.0
}
}
}
]
}
}
}
part of the response:
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 4,
"successful": 4,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 126.127914,
"hits": [
{
"_shard": "[xxxx][0]",
"_node": "EtnSgE81RjWc8x0YPcxK8Q",
"_index": "xxxx",
"_type": "_doc",
"_id": "827",
"_score": 126.127914,
"_source": {
"sku_id": 827,
"brand_good_id": 472201371,
"brand_good_name": "Peninsula Door Panels w/o TK",
"sku_name": "Peninsula Door Panels w/o TK",
"sku_goods_code": "PBDDP30",
"deleted": false,
"category_id": 1193295,
"visible": true,
},
"_explanation": {
"value": 126.127914,
"description": "sum of:",
"details": [
{
"value": 126.127914,
"description": "weight(sku_name:\"l ls (sel sul shelf st s squ singl su sus sh squar sq si sin stan susa start squa sa stom sink san sto stand she sk sta star susan sula ssed se ss sse shel sing sed set)\" in 103) [PerFieldSimilarity], result of:",
"details": [
{
"value": 126.127914,
"description": "score(doc=103,freq=1.0 = phraseFreq=1.0\n), product of:",
"details": [
{
"value": 157.94911,
"description": "idf(), sum of:",
"details": [
{
"value": 0.14261411,
"description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
"details": [
{
"value": 616,
"description": "docFreq",
"details": []
},
{
"value": 710,
"description": "docCount",
"details": []
}
]
},
{
"value": 4.861924,
"description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
"details": [
{
"value": 5,
"description": "docFreq",
"details": []
},
{
"value": 710,
"description": "docCount",
"details": []
}
]
},
{
"value": 5.6503816,
"description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
"details": [
{
"value": 2,
"description": "docFreq",
"details": []
},
{
"value": 710,
"description": "docCount",
"details": []
}
]
},
{
"value": 2.296975,
"description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
"details": [
{
"value": 71,
"description": "docFreq",
"details": []
},
{
"value": 710,
"description": "docCount",
"details": []
}
]
},
The full response is in (es/response.json at main · zegging/es · GitHub)
i run a python script to check, it seems like every prefix expansion count in the document idf(). It make the document have a very high score, which high than some sku like "LSSDOOR". I find nothing in elasticsearch document.
Does anyone know why match_phrase_prefix calculates document scores this way? I'd really appreciate any insights.
the script is:
import pandas as pd
import re
data = pd.read_json("response.json")
prefix_items_str = data["hits"]["hits"][0]["_explanation"]["details"][0]["description"]
matches = re.findall(r'"(.*?)"', prefix_items_str)[0].split(" ")
print(matches)
details = data["hits"]["hits"][0]["_explanation"]["details"][0]["details"][0]["details"][0]["details"]
for item in details:
index = details.index(item)
print(f"{matches[index]}: \t\t {item['value']}")
value_sum = sum(item['value'] for item in details)
print("\nvalue：", value_sum)