Hi Team I'm trying to understand which similarity algorithm is best suited for our use-case: 1.TF-IDF(default) 2.BM25 I've created two indices with mappings of both algorithms, but when I query, two indices are giving same documents with same score. Mappings for Index 1:(default algorithm) { "singleindex": { "aliases": {}, "mappings": { "people": { "properties": { "Application-Name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "Author": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "Character Count": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "Content-Type": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "Creation-Date": { "type": "date" }, "Last-Modified": { "type": "date" }, "Page-Count": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "Word-Count": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "content": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "meta:last-author": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "tika": { "properties": { "mime": { "properties": { "file": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } } } }, "xmpTPg:NPages": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } } }, "settings": { "index": { "creation_date": "1539597307747", "number_of_shards": "5", "number_of_replicas": "1", "uuid": "9jzKHct4T3qjI_EfFPFnzg", "version": { "created": "6020399" }, "provided_name": "singleindex" } } } } Mappings for Index 2:(BM25) { "tes_index": { "aliases": {}, "mappings": { "people": { "properties": { "Application-Name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "Author": { "type": "text", "fields": { "keyword": { "type": "keyword", "similarity": "BM25" } } }, "Character Count": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "Content-Type": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "Creation-Date": { "type": "date" }, "Last-Modified": { "type": "date" }, "Page-Count": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "Word-Count": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "content": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "meta:last-author": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "xmpTPg:NPages": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } } }, "settings": { "index": { "number_of_shards": "5", "provided_name": "tes_index", "similarity": { "default": { "type": "BM25" } }, "creation_date": "1539599395035", "number_of_replicas": "1", "uuid": "BFDAL1XuQ1O36KLIMDT2mw", "version": { "created": "6020399" } } } } } You can see in the settings. Please suggest what I'm missing here

Hello team Can someone please help.

Read this and specifically the "Also be patient" part. It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.

@dadoonet Thank you. I should have read this earlier. Will follow the guidelines strictly.

BM25 is already the default similarity. It has been since version 5.0. Before that, it used to be TF/IDF. If you want to use TF/IDF, you would have to configure an index to use the classic similarity. It may not be worth spending a lot of time testing TF/IDF though, as it has been deprecated.

Thanks @abdon So Elasticsearch by default uses BM25 ?

Okay Great. Thanks for your time @abdon Do I need to consider or Does ES supports Any algorithms which perform better than BM25?

"Better" is a very subjective term. There is a reason that Lucene and Elasticsearch use BM25 as the default: it seems to work well for a lot of use cases. Having said that, there are a number of other similarity algorithms available that may work better in certain use cases. You can find a list of all available similarities in the Elasticsearch documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html Maybe you find this video on our website interesting: https://www.elastic.co/elasticon/conf/2016/sf/improved-text-scoring-with-bm25 . It discusses the switch from TF/IDF to BM25.

@abdon Thanks for the reference links--helpful. What I observed is BM25 is better suited for most of the use cases. Though I didn't get how it calculates the probability of relevant documents from it's formula. Need to spend some time and should give it one more try. We are having long text fields more than 15 lines for each document. Is there any algorithm which suits for long text fields? If not it's good to go with BM25. Thanks for your time as always :slight_smile:

I would say that generally BM25 is great for longer fields. It is really designed for that full-text search use case. If you want to learn more about BM25, we have a great 3-part blog series on our website: [image] Practical BM25 - Part 1: How Shards Affect Relevance Scoring in Elasticsearch Similarity ranking (relevancy) in Elasticsearch relates directly to the amount of shards in your index. Learn more about how shards and relevancy are related, as well what you should consider when tuning search results. [image] Practical BM25 - Part 2: The BM25 Algorithm and its Variables BM25 is the default similarity ranking (relevancy) algorithm in Elasticsearch. Learn more about how it works by digging into the equation and exploring the concepts behind its variables. [image] Practical BM25 - Part 3: Considerations for Picking b and k1 in Elasticsearch Learn about best practices and other considerations before modifying the b and k1 values of the BM25 similarity ranking (relevancy) algorithm used by Elasticsearch.

Elasticseach: Default Similairty Algorithm and BM25 giving same results

Elastic Stack Elasticsearch

abdon (Abdon Pijpelink) October 17, 2018, 6:29am 11

I would say that generally BM25 is great for longer fields. It is really designed for that full-text search use case.

If you want to learn more about BM25, we have a great 3-part blog series on our website:

Topic		Replies	Views
Change default similarity to BM25 for all fields Elasticsearch	2	869	June 4, 2013
Default similarity in Elasticsearch Elasticsearch	5	867	October 27, 2016
What is default similarity in Elasticsearch 5 Elasticsearch	3	1101	December 8, 2016
To Interchange the Similarity Algorithm Elasticsearch	2	579	August 28, 2013
What is the path to utilize BM25F with Elastic Search Elasticsearch	5	1883	May 3, 2011

Elasticseach: Default Similairty Algorithm and BM25 giving same results

Related topics