Different score in 5.x compared to 1.x

[Akhilesh_Anb]

Hi,

I have upgraded elasticsearch cluster from 1.7 to 5.3.1

We have a query in 1.7 as below

{
"size": 100,
"query": {
"filtered": {
"query": {
"query_string": {
"query": "Akhilesh Anb"
}
},
"filter": {
"term": {
"show_id": 1
}
}
}
},
"sort": [
"_score",
{
"episode_version.episode.air_date": {
"order": "desc"
}
},
{
"tape_date": {
"order": "desc"
}
}
]
}

I changed that query to work in 5.3.1 as

{
"size": 100,
"query": {
"bool": {
"must": {
"query_string": {
"query": "Akhilesh Anb"
}
},
"filter": {
"term": {
"show_id": 1
}
}
}
},
"sort": [
"_score",
{
"episode_version.episode.air_date": {
"order": "desc"
}
},
{
"tape_date": {
"order": "desc"
}
}
]
}

The query in 5.3.1 is returning different score which leads to different order but returned results are as expected.

Please help me to solve this scoring problem.

Thanks,
Akhilesh Appana

[dadoonet (David Pilato)]

May be the scoring algorithm which is now bm25?

Read: https://www.elastic.co/guide/en/elasticsearch/reference/5.4/index-modules-similarity.html

[Akhilesh_Anb]

Hi dadoonet,

Thanks for very quick response.

So i need to update the index settings with type "BM25"?

Thanks,
Akhilesh Appana

[dadoonet (David Pilato)]

No. It's already BM25 in 5.x.
If you want to switch back to TF/IDF you need to use classic but BM25 is known to work pretty well.

[Akhilesh_Anb]

Thanks for explanation dadoonet.

I'll update the index settings as

PUT /my_index/_settings
{
"settings": {
"index": {
"similarity": {
"default": {
"type": "classic"
}
}
}
}
}

I'll update the story with the output whether this solved my problem or not. Until then please don't close this.

thanks,
Akhilesh Appana

[Akhilesh_Anb]

Hi,

I changed the similarity to classic. After changing the similarity i found results are better than before.

I was comparing the search results on Production (2.x) and Staging (5.x) environment. Search results looks same on both the environments, but still there is little difference in the search order (5.x). I am not sure why this is happening and can we achieve the exact search results order with the different ES versions?

thanks,
Akhilesh Appana