I have a strange issue with the _search API. I saw other people in this forum which had similar issues and the problem for them was that they had negative values. However in my case I am using dense vectors. Moreover my script_score is trying to calculate the cosine and I thought that cosine was returning negative values, however I am adding 1.0 to the score to make sure that it will never be negative.
Just to make sure that my dense vector don't have any issues I tried to do the dot product manually and it works. But when I try to use the cosineSimilarity with ES I get the following error.
Elasticsearch version : 7.4.2
lucene_version: 8.2.0
java version: 1.8.0
I have an index called "my_index" where it contains a field called embeddings. Its datatype is the following:
"embeddings": {
"type": "dense_vector",
"dims": 300
},
Every singe item contains a vector and its not null
GET my_index/_search
{
"query": {
"script_score": {
"query" : {
"match_all": {}
},
"script": {
"source": "cosineSimilarity(params.queryVector, doc['embeddings']) + 1.0",
"params": {
"queryVector": "[my_300_Dimenion_Vector]"
}
}
}
}
}
The error looks like this:
{
"error": {
"root_cause": [
{
"type": "exception",
"reason": "script score query returned an invalid score: NaN for doc: 9"
},
{
"type": "exception",
"reason": "script score query returned an invalid score: NaN for doc: 178"
},
{
"type": "exception",
"reason": "script score query returned an invalid score: NaN for doc: 24"
},
{
"type": "exception",
"reason": "script score query returned an invalid score: NaN for doc: 44"
},
{
"type": "exception",
"reason": "script score query returned an invalid score: NaN for doc: 60"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "webcontent_emb",
"node": "eIrwXaCYRrWjjOdUJkW66A",
"reason": {
"type": "exception",
"reason": "script score query returned an invalid score: NaN for doc: 9"
}
},
{
"shard": 1,
"index": "webcontent_emb",
"node": "aTC9SRxaRIK4gPdqCpEQ-w",
"reason": {
"type": "exception",
"reason": "script score query returned an invalid score: NaN for doc: 178"
}
},
{
"shard": 2,
"index": "webcontent_emb",
"node": "eIrwXaCYRrWjjOdUJkW66A",
"reason": {
"type": "exception",
"reason": "script score query returned an invalid score: NaN for doc: 24"
}
},
{
"shard": 3,
"index": "webcontent_emb",
"node": "aTC9SRxaRIK4gPdqCpEQ-w",
"reason": {
"type": "exception",
"reason": "script score query returned an invalid score: NaN for doc: 44"
}
},
{
"shard": 4,
"index": "webcontent_emb",
"node": "jsHhvQSzRO-zkM--fH1JdA",
"reason": {
"type": "exception",
"reason": "script score query returned an invalid score: NaN for doc: 60"
}
}
]
},
"status": 500
}