Inconsistent search scoring between shards

[Pyppe]

We have reindexed one big index [shards=1, size=220GB, documents=110M] to multiple nodes (9 shards & 3 nodes) using Elasticsearch 6.5.4 with java 1.8.0_191

However, now we are seeing that when doing a search the scoring between shards is not consistent (not 100% sure, but it might only be when shards are on different nodes). Identical documents (given the search criteria) can have quite different scores. Is this really expected behavior? How can we have consistent results?

Example query:

{
  "query": {
    "bool": {
      "must": [
        {
          "bool": {
            "should": [
              {
                "match_phrase": {
                  "searchableName.exact": {
                    "query": "example",
                    "boost": 3.0
                  }
                }
              },
              {
                "match_phrase": {
                  "searchableName": {
                    "query": "example",
                    "boost": 2.0
                  }
                }
              },
              {
                "prefix": {
                  "searchableName": {
                    "value": "example",
                    "boost": 1.45
                  }
                }
              },
              {
                "prefix": {
                  "searchableName.lower_latin": {
                    "value": "example",
                    "boost": 1.45
                  }
                }
              },
              {
                "prefix": {
                  "searchableName.reverse_lower_latin": {
                    "value": "elpmaxe",
                    "boost": 1.0
                  }
                }
              }
            ]
          }
        },
        {
          "bool": {
            "should": [
              {
                "term": {
                  "status": {
                    "value": "Valid",
                    "boost": 0.9
                  }
                }
              },
              {
                "term": {
                  "status": {
                    "value": "Pending",
                    "boost": 0.09
                  }
                }
              },
              {
                "term": {
                  "status": {
                    "value": "GracePeriod",
                    "boost": 0.009
                  }
                }
              },
              {
                "term": {
                  "status": {
                    "value": "Expired",
                    "boost": 0.0009
                  }
                }
              },
              {
                "term": {
                  "status": {
                    "value": "Invalid",
                    "boost": 0.00009
                  }
                }
              },
              {
                "term": {
                  "status": {
                    "value": "Unknown",
                    "boost": 0.000009
                  }
                }
              }
            ]
          }
        }
      ]
    }
  }
}

Example of two hits with different scores: https://pastebin.com/0TU4ngPc

[Pyppe]

Reading the documentation (https://www.elastic.co/guide/en/elasticsearch/guide/current/scoring-theory.html) it seems that this might kind of be the expected behavior (due to TF/IDF). And probably ignoring them (https://www.elastic.co/guide/en/elasticsearch/guide/current/ignoring-tfidf.html) might be the thing to do here in our case.

[Mark_Harwood]

See https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-search-type.html#dfs-query-then-fetch

[Pyppe]

Hmm. Setting it

{
  "search_type": "dfs_query_then_fetch",
  "query": {
    "match_all": {}
  }
}

gives me an error:

{
  "error": {
    "root_cause": [
      {
        "type": "parsing_exception",
        "reason": "Unknown key for a VALUE_STRING in [search_type].",
        "line": 2,
        "col": 18
      }
    ],
    "type": "parsing_exception",
    "reason": "Unknown key for a VALUE_STRING in [search_type].",
    "line": 2,
    "col": 18
  },
  "status": 400
}

[Mark_Harwood]

It's a setting passed as a URL parameter to the REST api as opposed to a field in the JSON body

[Pyppe]

Thanks so much, that did the trick!

PS. I find it a bit confusing that there are three exceptions (search_type , request_cache & allow_partial_search_results) that are not allowed to be part of the body, but instead only query-string. I'd argue you should be able to use only the body when defining the search criteria. E.g. testability would be easier if there were single JSON-structure that would define the criteria as a whole.

Exceptions are bad, mmkay?

[Mark_Harwood]

I'm not 100% clear on the rationale myself but a part of it may be that the software channeling search requests (e.g. Kibana) can introduce application-wide policies like search timeouts by setting these headers in a single place and not tampering with the contents of the request bodies which can vary from request to request and come from many different parts of the application.

[Pyppe]

That's a valid argument, I guess. But why not then allow them in both query-string and body? (And next we could argue which would then take the precedence if both exist... )

But anyway, thanks again for letting me know about the search_type parameter.