Why is idf different for same term in same field in same shard?

Elastic Stack Elasticsearch
1

I am using the _explain API to understand how 2 different documents ended up having different scores. I am surprised to see that for the same field and term, the idf for that term is different. The number of shards for this index is 1.

Here are the excerpts:

doc 1:

                    "value": 0.8311164,
                    "description": "weight(description:temperatur in 36) [PerFieldSimilarity], result of:",
                    "details": [
                      {
                        "value": 0.8311164,
                        "description": "score(doc=36,freq=1.0), product of:",
                        "details": [
                          {
                            "value": 0.3390907,
                            "description": "queryWeight, product of:",
                            "details": [
                              {
                                "value": 3.9216242,
                                "description": "idf(docFreq=6, maxDocs=130)",
                                "details": []
                              },
                              {
                                "value": 0.0864669,
                                "description": "queryNorm",
                                "details": []
                              }
                            ]
                          },

..........................

doc2

                    "value": 0.6850641,
                    "description": "weight(description:temperatur in 57) [PerFieldSimilarity], result of:",
                    "details": [
                      {
                        "value": 0.6850641,
                        "description": "score(doc=57,freq=1.0), product of:",
                        "details": [
                          {
                            "value": 0.3556344,
                            "description": "queryWeight, product of:",
                            "details": [
                              {
                                "value": 3.8526313,
                                "description": "idf(docFreq=5, maxDocs=104)",
                                "details": []
                              },
                              {
                                "value": 0.09230948,
                                "description": "queryNorm",
                                "details": []
                              }
                            ]
                          },

...................................

From "Relevance is broken", I understood that having multiple shards can cause this idf discrepancy. But, here I have only 1 shard.

As you can see from the excerpts, the maxDocs is different in both. How is it possible?

2

Having the same problem here.
I'm actually using different shards, but I'm setting both dfs_query_then_fetch and preference; from my understanding it should be enough to avoid that issue according the the doc.

3

Hi,
both of you, probably, have the multi match query with cross_fields type, smth like this

" multi_match": {
"query": "smth to search",
"type": "cross_fields",
"fields": ["name^10", "name_keywords^1.5", "description^0.7", "attr.text^0.7"]
}

In which case there's already an issue for that:
https://github.com/elastic/elasticsearch/issues/10346

Hope this helps and the issue will be resolved soon enough :slight_smile: