# Inconsistent search scoring between shards **URL:** \ **Category:** Elasticsearch\ **Created:** [January 10, 2019, 9:18am UTC](https://discuss.elastic.co/t/inconsistent-search-scoring-between-shards/163686 "2019-01-10T09:18:26Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Pyppe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pyppe/32/3531_2.png) [@Pyppe](https://discuss.elastic.co/u/Pyppe)\ **Post date:** [January 10, 2019, 9:18am UTC](https://discuss.elastic.co/t/inconsistent-search-scoring-between-shards/163686/1 "2019-01-10T09:18:26Z")
We have reindexed one big index [shards=1, size=220GB, documents=110M] to multiple nodes (9 shards & 3 nodes) using `Elasticsearch 6.5.4` with `java 1.8.0_191` However, now we are seeing that when doing a search the scoring between shards is not consistent (not 100% sure, but it might only be when shards are on different nodes). Identical documents (given the search criteria) can have quite different scores. Is this really expected behavior? How can we have consistent results? Example query: ```json { "query": { "bool": { "must": [ { "bool": { "should": [ { "match_phrase": { "searchableName.exact": { "query": "example", "boost": 3.0 } } }, { "match_phrase": { "searchableName": { "query": "example", "boost": 2.0 } } }, { "prefix": { "searchableName": { "value": "example", "boost": 1.45 } } }, { "prefix": { "searchableName.lower_latin": { "value": "example", "boost": 1.45 } } }, { "prefix": { "searchableName.reverse_lower_latin": { "value": "elpmaxe", "boost": 1.0 } } } ] } }, { "bool": { "should": [ { "term": { "status": { "value": "Valid", "boost": 0.9 } } }, { "term": { "status": { "value": "Pending", "boost": 0.09 } } }, { "term": { "status": { "value": "GracePeriod", "boost": 0.009 } } }, { "term": { "status": { "value": "Expired", "boost": 0.0009 } } }, { "term": { "status": { "value": "Invalid", "boost": 0.00009 } } }, { "term": { "status": { "value": "Unknown", "boost": 0.000009 } } } ] } } ] } } } ``` Example of two hits with different scores: [https://pastebin.com/0TU4ngPc](https://pastebin.com/0TU4ngPc) ---
**Author:** ![Pyppe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pyppe/32/3531_2.png) [@Pyppe](https://discuss.elastic.co/u/Pyppe)\ **Post date:** [January 10, 2019, 9:51am UTC](https://discuss.elastic.co/t/inconsistent-search-scoring-between-shards/163686/2 "2019-01-10T09:51:31Z")
Reading the documentation ([https://www.elastic.co/guide/en/elasticsearch/guide/current/scoring-theory.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/scoring-theory.html)) it seems that this might kind of be the expected behavior (due to TF/IDF). And probably ignoring them ([https://www.elastic.co/guide/en/elasticsearch/guide/current/ignoring-tfidf.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/ignoring-tfidf.html)) might be the thing to do here in our case. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [January 10, 2019, 10:18am UTC](https://discuss.elastic.co/t/inconsistent-search-scoring-between-shards/163686/3 "2019-01-10T10:18:01Z")
See [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-search-type.html#dfs-query-then-fetch](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-search-type.html#dfs-query-then-fetch) ---
**Author:** ![Pyppe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pyppe/32/3531_2.png) [@Pyppe](https://discuss.elastic.co/u/Pyppe)\ **Post date:** [January 10, 2019, 1:22pm UTC](https://discuss.elastic.co/t/inconsistent-search-scoring-between-shards/163686/4 "2019-01-10T13:22:46Z")
Hmm. Setting it ```json { "search_type": "dfs_query_then_fetch", "query": { "match_all": {} } } ``` gives me an error: ```json { "error": { "root_cause": [ { "type": "parsing_exception", "reason": "Unknown key for a VALUE_STRING in [search_type].", "line": 2, "col": 18 } ], "type": "parsing_exception", "reason": "Unknown key for a VALUE_STRING in [search_type].", "line": 2, "col": 18 }, "status": 400 } ``` ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [January 10, 2019, 1:25pm UTC](https://discuss.elastic.co/t/inconsistent-search-scoring-between-shards/163686/5 "2019-01-10T13:25:04Z")
It's a setting passed as a URL parameter to the REST api as opposed to a field in the JSON body ---
**Author:** ![Pyppe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pyppe/32/3531_2.png) [@Pyppe](https://discuss.elastic.co/u/Pyppe)\ **Post date:** [January 10, 2019, 3:47pm UTC](https://discuss.elastic.co/t/inconsistent-search-scoring-between-shards/163686/6 "2019-01-10T15:47:51Z")
Thanks so much, that did the trick! 🙌 PS. I find it a bit confusing that there are three exceptions (`search_type` , `request_cache` & `allow_partial_search_results`) that are not allowed to be part of the body, but instead only query-string. I'd argue you should be able to use only the body when defining the search criteria. E.g. testability would be easier if there were single JSON-structure that would define the criteria as a whole. Exceptions are bad, mmkay? 🙂 ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [January 10, 2019, 3:53pm UTC](https://discuss.elastic.co/t/inconsistent-search-scoring-between-shards/163686/7 "2019-01-10T15:53:10Z")
I'm not 100% clear on the rationale myself but a part of it may be that the software channeling search requests (e.g. Kibana) can introduce application-wide policies like search timeouts by setting these headers in a single place and not tampering with the contents of the request bodies which can vary from request to request and come from many different parts of the application. ---
**Author:** ![Pyppe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pyppe/32/3531_2.png) [@Pyppe](https://discuss.elastic.co/u/Pyppe)\ **Post date:** [January 10, 2019, 4:00pm UTC](https://discuss.elastic.co/t/inconsistent-search-scoring-between-shards/163686/8 "2019-01-10T16:00:24Z")
That's a valid argument, I guess. But why not then allow them in both query-string and body? (And next we could argue which would then take the precedence if both exist... 😉) But anyway, thanks again for letting me know about the `search_type` parameter. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 7, 2019, 4:00pm UTC](https://discuss.elastic.co/t/inconsistent-search-scoring-between-shards/163686/9 "2019-02-07T16:00:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.