# Order of results inconsistent between similar indices and same data set **URL:** \ **Category:** Elasticsearch\ **Created:** [February 7, 2020, 4:10pm UTC](https://discuss.elastic.co/t/order-of-results-inconsistent-between-similar-indices-and-same-data-set/218381 "2020-02-07T16:10:17Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\ **Post date:** [February 7, 2020, 4:10pm UTC](https://discuss.elastic.co/t/order-of-results-inconsistent-between-similar-indices-and-same-data-set/218381/1 "2020-02-07T16:10:17Z")
I have 2 identical indices with similar (overlapping) data though one index has more data. I am getting the same documents in each index with the same query with a different order. When I search for "foo" (boost 2) and "bar" I expect to get documents with "foo" and "bar" before documents with just "foo". In Index 1 this works correctly and I get "some foo blah bar document" as the first result. In Index 2 I get "foo some" as the first result and the desired result "some foo blah bar" is waaaay down the order with a lower score. I want to understand or rather influence the score: more occurrences must always score higher in our scenario. Can anyone explain the "wrong" result below? Can we determine/change the algorithm for scoring? ## Query (same for both indices) ``` { "from": 0, "size": 1000, "_source": ["text_en"], "explain": true, "query": { "bool": { "should": [ { "match": { "text_en": { "query": "foo", "boost": 2 } } }, { "match": { "text_en": "bar" } } ], "minimum_should_match": 1 } } } ``` ## Result from Index 1 (correct) Here we get our document `"some foo 205 x bar"` as the first result with the highest score: ``` "_score": 19.79837, "_source": { "text_en": "some foo 205 x bar" }, "_explanation": { "value": 19.79837, "description": "sum of:", "details": [ { "value": 14.814142, "description": "weight(text_en:foo in 31366) [PerFieldSimilarity], result of:", "details": [ { "value": 14.814142, "description": "score(freq=1.0), product of:", "details": [ { "value": 4.4, "description": "boost", "details": [] }, { "value": 8.928385, "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:", "details": [ { "value": 195, "description": "n, number of documents containing term", "details": [] }, { "value": 1474669, "description": "N, total number of documents with field", "details": [] } ] }, { "value": 0.37709513, "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:", "details": [ { "value": 1.0, "description": "freq, occurrences of term within document", "details": [] }, { "value": 1.2, "description": "k1, term saturation parameter", "details": [] }, { "value": 0.75, "description": "b, length normalization parameter", "details": [] }, { "value": 29.0, "description": "dl, length of field", "details": [] }, { "value": 19.306864, "description": "avgdl, average length of field", "details": [] } ] } ] } ] }, { "value": 4.9842277, "description": "weight(text_en:bar in 31366) [PerFieldSimilarity], result of:", "details": [ { "value": 4.9842277, "description": "score(freq=1.0), product of:", "details": [ { "value": 2.2, "description": "boost", "details": [] }, { "value": 6.0079217, "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:", "details": [ { "value": 3626, "description": "n, number of documents containing term", "details": [] }, { "value": 1474669, "description": "N, total number of documents with field", "details": [] } ] }, { "value": 0.37709513, "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:", "details": [ { "value": 1.0, "description": "freq, occurrences of term within document", "details": [] }, { "value": 1.2, "description": "k1, term saturation parameter", "details": [] }, { "value": 0.75, "description": "b, length normalization parameter", "details": [] }, { "value": 29.0, "description": "dl, length of field", "details": [] }, { "value": 19.306864, "description": "avgdl, average length of field", "details": [] } ] } ] } ] } ] ``` ## Result from Index 2 Here the wrong document `"some foo"` get's a higher score (21) than our desired document (19) ``` "_score": 21.883192, "_source": { "text_en": "some foo", }, "_explanation": { "value": 21.883192, "description": "sum of:", "details": [ ``` The explanation is similar to above. ---
**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\ **Post date:** [February 10, 2020, 7:12am UTC](https://discuss.elastic.co/t/order-of-results-inconsistent-between-similar-indices-and-same-data-set/218381/2 "2020-02-10T07:12:12Z")
I've narrowed the problem down to an unusually high TF score for the document `some foo`. The TF is calculated as `0.5993144` which is twice as high as the TF for `some foo bar`. > tf, computed as freq / (freq + k1 \* (1 - b + b \* dl / avgdl)) from: It looks to me as though `dl` (dl, length of field) is incorrectly calculated as `2.0`. If my source field is `"text_en": "sssssssssss ffffffffffffff",` how does it get `2.0`?? It should be `26`. Note: "foo" was just a censored version of our actual search term which was 14 characters. I've shown this below as "ffffffffffffff" and "some" as "sssssssssss" to get the correct lengths. ``` "_score": 21.883192, "_source": { "text_en": "sssssssssss ffffffffffffff", }, "_explanation": { "value": 21.883192, "description": "sum of:", "details": [ { "value": 21.883192, "description": "weight(text_en:ffffffffffffff in 289853) [PerFieldSimilarity], result of:", "details": [ { "value": 21.883192, "description": "score(freq=1.0), product of:", "details": [ { "value": 4.4, "description": "boost", "details": [] }, { "value": 8.298571, "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:", "details": [ { "value": 319, "description": "n, number of documents containing term", "details": [] }, { "value": 1283790, "description": "N, total number of documents with field", "details": [] } ] }, { "value": 0.5993144, "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:", "details": [ { "value": 1.0, "description": "freq, occurrences of term within document", "details": [] }, { "value": 1.2, "description": "k1, term saturation parameter", "details": [] }, { "value": 0.75, "description": "b, length normalization parameter", "details": [] }, { "value": 2.0, "description": "dl, length of field", "details": [] }, { "value": 4.8836966, "description": "avgdl, average length of field", "details": [] } ] } ] } ] } ] } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 9, 2020, 7:12am UTC](https://discuss.elastic.co/t/order-of-results-inconsistent-between-similar-indices-and-same-data-set/218381/3 "2020-03-09T07:12:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.