# Elasticseach: Default Similairty Algorithm and BM25 giving same results

**URL:** <https://discuss.elastic.co/t/elasticseach-default-similairty-algorithm-and-bm25-giving-same-results/152470>\
**Category:** Elasticsearch\
**Created:** [October 15, 2018, 10:31am UTC](https://discuss.elastic.co/t/elasticseach-default-similairty-algorithm-and-bm25-giving-same-results/152470 "2018-10-15T10:31:16Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [October 15, 2018, 10:31am UTC](https://discuss.elastic.co/t/elasticseach-default-similairty-algorithm-and-bm25-giving-same-results/152470/1 "2018-10-15T10:31:17Z")

</div>

Hi Team

I'm trying to understand which similarity algorithm is best suited for our use-case:  
1.TF-IDF(default)  
2.BM25

I've created two indices with mappings of both algorithms, but when I query, two indices are giving same documents with same score.

Mappings for Index 1:(default algorithm)

> ```
> {
> "singleindex": {
> "aliases": {},
> "mappings": {
> "people": {
> "properties": {
> "Application-Name": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "Author": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "Character Count": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "Content-Type": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "Creation-Date": {
> "type": "date"
> },
> "Last-Modified": {
> "type": "date"
> },
> "Page-Count": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "Word-Count": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "content": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "meta:last-author": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "tika": {
> "properties": {
> "mime": {
> "properties": {
> "file": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> }
> }
> }
> }
> },
> "xmpTPg:NPages": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> }
> }
> }
> },
> "settings": {
> "index": {
> "creation_date": "1539597307747",
> "number_of_shards": "5",
> "number_of_replicas": "1",
> "uuid": "9jzKHct4T3qjI_EfFPFnzg",
> "version": {
> "created": "6020399"
> },
> "provided_name": "singleindex"
> }
> }
> }
> }
> 
> ```

Mappings for Index 2:(BM25)

```
   {
  "tes_index": {
    "aliases": {},
    "mappings": {
      "people": {
        "properties": {
          "Application-Name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "Author": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "similarity": "BM25"
              }
            }
          },
          "Character Count": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "Content-Type": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "Creation-Date": {
            "type": "date"
          },
          "Last-Modified": {
            "type": "date"
          },
          "Page-Count": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "Word-Count": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "content": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "meta:last-author": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "xmpTPg:NPages": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "number_of_shards": "5",
        "provided_name": "tes_index",
        "similarity": {
          "default": {
            "type": "BM25"
          }
        },
        "creation_date": "1539599395035",
        "number_of_replicas": "1",
        "uuid": "BFDAL1XuQ1O36KLIMDT2mw",
        "version": {
          "created": "6020399"
        }
      }
    }
  }
}

```

You can see in the settings. Please suggest what I'm missing here

---

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [October 15, 2018, 3:37pm UTC](https://discuss.elastic.co/t/elasticseach-default-similairty-algorithm-and-bm25-giving-same-results/152470/2 "2018-10-15T15:37:32Z")

</div>

Hello team

Can someone please help.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 15, 2018, 3:51pm UTC](https://discuss.elastic.co/t/elasticseach-default-similairty-algorithm-and-bm25-giving-same-results/152470/3 "2018-10-15T15:51:04Z")

</div>

Read [this](https://discuss.elastic.co/t/about-the-elasticsearch-category/21) and specifically the "Also be patient" part.

It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.

---

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [October 15, 2018, 4:38pm UTC](https://discuss.elastic.co/t/elasticseach-default-similairty-algorithm-and-bm25-giving-same-results/152470/4 "2018-10-15T16:38:36Z")

</div>

@dadoonet  
Thank you. I should have read this earlier.

Will follow the guidelines strictly.

---

<div class="post-metadata">

**Author:** ![abdon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdon/32/9195_2.png) [@abdon](https://discuss.elastic.co/u/abdon)\
**Post date:** [October 16, 2018, 5:24am UTC](https://discuss.elastic.co/t/elasticseach-default-similairty-algorithm-and-bm25-giving-same-results/152470/5 "2018-10-16T05:24:41Z")

</div>

BM25 is already the default similarity. It has been since version 5.0. Before that, it used to be TF/IDF. If you want to use TF/IDF, you would have to configure an index to use the `classic` similarity.

It may not be worth spending a lot of time testing TF/IDF though, as it [has been deprecated](https://github.com/elastic/elasticsearch/issues/23208).

---

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [October 16, 2018, 8:19am UTC](https://discuss.elastic.co/t/elasticseach-default-similairty-algorithm-and-bm25-giving-same-results/152470/6 "2018-10-16T08:19:59Z")

</div>

Thanks @abdon  
So Elasticsearch by default uses BM25 ?

---

<div class="post-metadata">

**Author:** ![abdon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdon/32/9195_2.png) [@abdon](https://discuss.elastic.co/u/abdon)\
**Post date:** [October 16, 2018, 8:35am UTC](https://discuss.elastic.co/t/elasticseach-default-similairty-algorithm-and-bm25-giving-same-results/152470/7 "2018-10-16T08:35:33Z")

</div>

Yes, since version 5.0

---

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [October 16, 2018, 9:04am UTC](https://discuss.elastic.co/t/elasticseach-default-similairty-algorithm-and-bm25-giving-same-results/152470/8 "2018-10-16T09:04:17Z")

</div>

Okay Great.

Thanks for your time @abdon

Do I need to consider or Does ES supports Any algorithms which perform better than BM25?

---

<div class="post-metadata">

**Author:** ![abdon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdon/32/9195_2.png) [@abdon](https://discuss.elastic.co/u/abdon)\
**Post date:** [October 16, 2018, 6:28pm UTC](https://discuss.elastic.co/t/elasticseach-default-similairty-algorithm-and-bm25-giving-same-results/152470/9 "2018-10-16T18:28:23Z")

</div>

"Better" is a very subjective term. There is a reason that Lucene and Elasticsearch use BM25 as the default: it seems to work well for a lot of use cases. Having said that, there are a number of other similarity algorithms available that may work better in certain use cases. You can find a list of all available similarities in the Elasticsearch documentation: [https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html)

Maybe you find this video on our website interesting: [https://www.elastic.co/elasticon/conf/2016/sf/improved-text-scoring-with-bm25](https://www.elastic.co/elasticon/conf/2016/sf/improved-text-scoring-with-bm25) . It discusses the switch from TF/IDF to BM25.

---

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [October 17, 2018, 6:21am UTC](https://discuss.elastic.co/t/elasticseach-default-similairty-algorithm-and-bm25-giving-same-results/152470/10 "2018-10-17T06:21:14Z")

</div>

@abdon Thanks for the reference links--helpful. What I observed is BM25 is better suited for most of the use cases. Though I didn't get how it calculates the probability of relevant documents from it's formula. Need to spend some time and should give it one more try.

We are having long text fields more than 15 lines for each document. Is there any algorithm which suits for long text fields? If not it's good to go with BM25.

Thanks for your time as always 🙂

---

<div class="post-metadata">

**Author:** ![abdon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdon/32/9195_2.png) [@abdon](https://discuss.elastic.co/u/abdon)\
**Post date:** [October 17, 2018, 6:29am UTC](https://discuss.elastic.co/t/elasticseach-default-similairty-algorithm-and-bm25-giving-same-results/152470/11 "2018-10-17T06:29:14Z")

</div>

I would say that generally BM25 is great for longer fields. It is really designed for that full-text search use case.

If you want to learn more about BM25, we have a great 3-part blog series on our website:

> **[Practical BM25 - Part 1: How Shards Affect Relevance Scoring in Elasticsearch](https://www.elastic.co/blog/practical-bm25-part-1-how-shards-affect-relevance-scoring-in-elasticsearch)**
>
> Similarity ranking (relevancy) in Elasticsearch relates directly to the amount of shards in your index. Learn more about how shards and relevancy are related, as well what you should consider when tuning search results.

> **[Practical BM25 - Part 2: The BM25 Algorithm and its Variables](https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables)**
>
> BM25 is the default similarity ranking (relevancy) algorithm in Elasticsearch. Learn more about how it works by digging into the equation and exploring the concepts behind its variables.

> **[Practical BM25 - Part 3: Considerations for Picking b and k1 in Elasticsearch](https://www.elastic.co/blog/practical-bm25-part-3-considerations-for-picking-b-and-k1-in-elasticsearch)**
>
> Learn about best practices and other considerations before modifying the b and k1 values of the BM25 similarity ranking (relevancy) algorithm used by Elasticsearch.

---

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [October 17, 2018, 6:31am UTC](https://discuss.elastic.co/t/elasticseach-default-similairty-algorithm-and-bm25-giving-same-results/152470/12 "2018-10-17T06:31:32Z")

</div>

@abdon

Good to hear. Thank you so much

will go through the blogs. that should help

Thanks again 🙂

-- Rahul

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 14, 2018, 6:31am UTC](https://discuss.elastic.co/t/elasticseach-default-similairty-algorithm-and-bm25-giving-same-results/152470/13 "2018-11-14T06:31:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.