Using painless scripting to re-implement BM25 scoring

Elastic Stack Elasticsearch
1

Hello everyone

I have indexed a few hundreds of millions of data into ElasticSearch using the default parameters of b and k1. It is prohibitive for me to reindex the data however i would like to optimize the parameters b and k1 of bm25 for better scoring.
To my understanding there are some functions in Painless scripting that could compute/fetch tf and idf scores of a token in a document.
Could you please reproduce BM25 in Painless scripting so that i could tune the b and k1 parameters?

Thank you all in advance
Dimitris

2

In Elasticsearch the module implementing textual scoring is called similarity. There isn't a need to write a painless script, since b and k1 can be customized. However if you are intent on using painless, you can write a similarity script.

3

Unfortunately this will not work.
As i mentioned i do not want to reindex.
I have hundreds of millions of data.
i just need to optimize the values of b and k1.
I just want to replicate bm25 with painless scripting so to just change thw values of b and k1 as needed.

Thank you again

4

The similarity can be changed on an existing index, no reindexing necessary. What do you see implying reindexing would be needed?

5

I thought that "PUT /index ..." is a creation of an index.
Dont i need to re-index the data once i change the mapping/settings ?

6

While some settings cannot be changed (eg number_of_shards) many settings can be, like the configured similarity. Use the update settings api.

7

Dont i need to re-index the data once i change the mapping/settings ?

Since the similarity settings are not baked into the index, changing these parameters does not requiring reindexing.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.