Query can return negative score? I couldn't find any spec on documentation. The minimum value of score is not 0?

Elastic Stack Elasticsearch
1

I got some negative score for document -1.0910728

I didn't expect this, query can return negative score?
Should I use min_score? or is this just a bug?

  • Update: min_score seems not helped, because I want use function score, and function_score seems calculate score before min_score.

and the negative scored document has matched keyword, and have highlights also!

below is the query I used.

{
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "must": [
            {
              "multi_match": {
                "query": "hello",
                "type": "cross_fields",
                "fields": [
                  "introduction.text^3",
                     ......                  
                ],
                "slop": 10
              }
            }
          ],
          "filter": [
            {
              "range": {
                "age": {
                  "gte": 1
                }
              }
            },
            {
              "range": {
                "age": {
                  "lte": 1
                }
              }
            }
          ]
        }
      },
      "boost": 5,
      "score_mode": "multiply",
      "boost_mode": "multiply",
      "functions": [  ]
    }
  },
  "from": 0,
  "size": 30,
  "sort": [
    "_score"
  ]
}
2

Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

A full reproduction script will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.

3

@dadoonet,
Thank you for your reply.

If I could, I would provide recreation script so that others can reproduce.
but in this case, I'm having hard time to reproduce the error.
I got the negative score, but how can I reproduce the negative score problem is different thing, I think.

I will update if I find the script. but until now, I didn't. and I'm trying.

By the way, Do you know the rule for score?
I thought score's minimum value is zero. but am I wrong?

I would be glad if I know the negative score is bug or not regardless of finding how reproduce the problem.

Thank you!

4

What do you have with "explain": true?

5

@dadoonet,

ah, I forgot the explain: true.
Thank you for remind me. This is what I got with explain: true

        "_explanation": {
          "value": -16.503696,
          "description": "sum of:",
          "details": [
            {
              "value": 0,
              "description": "max of:",
              "details": [
                {
                  "value": -16.503696,
                  "description": "weight(profile.occupation_name:개발 in 0) [PerFieldSimilarity], result of:",
                  "details": [
                    {
                      "value": -16.503696,
                      "description": "score(freq=1.0), product of:",
                      "details": [
                        {
                          "value": 55,
                          "description": "boost",
                          "details": []
                        },
                        {
                          "value": -0.6061358,
                          "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                          "details": [
                            {
                              "value": 5,
                              "description": "n, number of documents containing term",
                              "details": []
                            },
                            {
                              "value": 2,
                              "description": "N, total number of documents with field",
                              "details": []
                            }
                          ]
                        },
                        {
                          "value": 0.49504948,
                          "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                          "details": [
                            {
                              "value": 1,
                              "description": "freq, occurrences of term within document",
                              "details": []
                            },
                            {
                              "value": 1.2,
                              "description": "k1, term saturation parameter",
                              "details": []
                            },
                            {
                              "value": 0.75,
                              "description": "b, length normalization parameter",
                              "details": []
                            },
                            {
                              "value": 2,
                              "description": "dl, length of field",
                              "details": []
                            },
                            {
                              "value": 2.5,
                              "description": "avgdl, average length of field",
                              "details": []
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "value": 0,
              "description": "match on required clause, product of:",
              "details": [
                {
                  "value": 0,
                  "description": "# clause",
                  "details": []
                },
                {
                  "value": 1,
                  "description": "profile.tos_service:T",
                  "details": []
                }
              ]
            }
          ]
        }
      }
6

@jpountz Do you know why this is happening?

8

I can reproduce this situation with below script.
maybe the problem could be n is bigger than N
@dadoonet

curl -X DELETE "localhost:9700/some_index"

curl -X PUT "localhost:9700/some_index" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  },
  "mappings": {
    "dynamic": false,
    "properties": {
      "user_id": {
        "type": "integer"
      },
      "introduction": {
        "type": "text"
      },
      "occupation_name": {
        "type": "text", 
        "fields": {
          "standard": { "type": "text" },  
          "raw": { "type": "keyword" }     
        }
      }
    }
  }
}
'


curl -X PUT "localhost:9700/some_index/_doc/1" -H "Content-Type: application/json" -d'
{
  "user_id": 1,
  "introduction": "ruby web developer"
}
'


curl -X PUT "localhost:9700/some_index/_doc/2" -H "Content-Type: application/json" -d'
{
  "user_id": 2,
  "introduction": "ruby developer",
  "occupation_name": "ruby developer"
}
'

curl -X PUT "localhost:9700/some_index/_doc/3" -H "Content-Type: application/json" -d'
{
  "user_id": 3,
  "introduction": "ruby and rails develop"
}
'

curl -X PUT "localhost:9700/some_index/_doc/4" -H "Content-Type: application/json" -d'
{
  "user_id": 4,
  "introduction": "I develop with ruby on rails"
}
'

curl -X PUT "localhost:9700/some_index/_doc/5" -H "Content-Type: application/json" -d'
{
  "user_id": 5,
  "introduction": "I develop with ruby on rails"
}
'

curl -X PUT "localhost:9700/some_index/_doc/6" -H "Content-Type: application/json" -d'
{
  "user_id": 6,
  "introduction": "I develop with ruby on rails"
}
'

curl -X PUT "localhost:9700/some_index/_doc/7" -H "Content-Type: application/json" -d'
{
  "user_id": 7,
  "introduction": "I develop with ruby on rails"
}
'

curl -X PUT "localhost:9700/some_index/_doc/8" -H "Content-Type: application/json" -d'
{
  "user_id": 8,
  "introduction": "I develop with ruby on rails"
}
'

curl -X PUT "localhost:9700/some_index/_doc/9" -H "Content-Type: application/json" -d'
{
  "user_id": 9,
  "introduction": "I develop with ruby on rails"
}
'

curl -X PUT "localhost:9700/some_index/_doc/10" -H "Content-Type: application/json" -d'
{
  "user_id": 10,
  "introduction": "I develop with ruby on rails"
}
'

curl -X PUT "localhost:9700/some_index/_doc/11" -H "Content-Type: application/json" -d'
{
  "user_id": 11,
  "introduction": "I develop with ruby on rails"
}
'

and query


curl -X GET "localhost:9700/some_index/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "must": [
            {
              "multi_match": {
                "query": "ruby",
                "type": "cross_fields",
                "fields": [
                  "occupation_name^5",
                  "introduction^2"                  
                ],
                "slop": 10
              }
            }
          ]
        }
      },
      "functions": []
    }
  },
  "from": 0,
  "size": 10,
  "explain": true
}
'

the response:

...
  "_source": {
    "user_id": 2,
    "introduction": "ruby developer",
    "occupation_name": "ruby developer"
  },
  "_explanation": {
    "value": 0.45840853,
    "description": "max of:",
    "details": [
      {
        "value": -1.1157178,
        "description": "weight(occupation_name:ruby in 0) [PerFieldSimilarity], result of:",
        "details": [
          {
            "value": -1.1157178,
            "description": "score(freq=1.0), product of:",
            "details": [
              {
                "value": 11,
                "description": "boost",
                "details": []
              },
              {
                "value": -0.22314355,
                "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                "details": [
                  {
                    "value": 2,
                    "description": "n, number of documents containing term",
                    "details": []
                  },
                  {
                    "value": 1,
                    "description": "N, total number of documents with field",
                    "details": []
                  }
                ]
              },
...
1 Like
9

Thanks for reporting this. It is a bug. I opened https://github.com/elastic/elasticsearch/issues/44700.

2 Likes
10

Thank you!

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.