Script Score Query Cosine Similarity

[angelazhao]

I'm following the example that calculates cosine similarity using sparse vectors found here: https://www.elastic.co/guide/en/elasticsearch/reference/7.x/query-dsl-script-score-query.html

My query is the following:
GET /my-index-name/_search
{
"query": {
"script_score": {
"query": {
"match_all": {}
},
"script": {
"source": "cosineSimilaritySparse(params.queryVector, doc['word_vector'])",
"params": {
"queryVector": {"11": 1.0, "1": 1.0, "12": 1.0, "5": 1.0, "7": 1.0, "0": 1.0, "4": 1.0, "8": 1.0, "21": 1.0, "6": 1.0, "19": 1.0, "14": 1.0, "245": 1.0, "940": 1.0, "105": 1.0, "416": 1.0}
}
}
}
}
}

After running this, I get a compile error. The full error looks like this:
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"cosineSimilaritySparse(pa ...",
"^---- HERE"
],
"script": "cosineSimilaritySparse(params.queryVector, doc['word_vector'])",
"lang": "painless"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "sample-bulk-1",
"node": "X-Dbh7dATn-2FLbxCBct5g",
"reason": {
"type": "query_shard_exception",
"reason": "script_score: the script could not be loaded",
"index_uuid": "HP84rvNyQjiRz4xdUWBZ7g",
"index": "sample-bulk-1",
"caused_by": {
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"cosineSimilaritySparse(pa ...",
"^---- HERE"
],
"script": "cosineSimilaritySparse(params.queryVector, doc['word_vector'])",
"lang": "painless",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Unknown call [cosineSimilaritySparse] with [2] arguments."
}
}
}
}
],
"caused_by": {
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"cosineSimilaritySparse(pa ...",
"^---- HERE"
],
"script": "cosineSimilaritySparse(params.queryVector, doc['word_vector'])",
"lang": "painless",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Unknown call [cosineSimilaritySparse] with [2] arguments."
}
}
},
"status": 400
}

[ghorne (Graham Horne)]

I am getting the exact same error trying to test cosine similarity with painless scripting and trying to get a dense vector score, using V7.1.1. somehow scripting seems to be turned off?

[mayya (Mayya Sharipova)]

@angelazhao @ghorne Our apologies, cosineSimilarity and dotProduct will be available from 7.3. In later releases, we are adding more functions such as L1norm (Manhattan distance) and L2norm ( Euclidean distance)

[ghorne (Graham Horne)]

many thanks @mayya , managed to get this working with snapshot build 8, would be really interested in getting this functionality working faster for similarity ranking. Tested with worst case 1024 Vectors and 10,000 V representations against docs...