Script_fields is throwing Exception

Elastic Stack Elasticsearch
1

Hi All,
Am having the data with mappings as shown:

"ymme_type": {
"_all": {
"auto_boost": true,
"index_analyzer": "wordAnalyzer",
"search_analyzer": "whitespace_analyzer"
},
"properties": {
"Engine": {
"type": "string",
"index": "not_analyzed"
},
"EngineCode": {
"type": "string",
"include_in_all": false
},
"Make": {
"type": "string",
"boost": 3,
"index": "not_analyzed",
"norms": {
"enabled": true
}
},
"MakeCode": {
"type": "string",
"include_in_all": false
},
"Model": {
"type": "string",
"boost": 2,
"index": "not_analyzed",
"norms": {
"enabled": true
}
},
"ModelCode": {
"type": "string",
"include_in_all": false
},
"ShortYear": {
"type": "string",
"boost": 4,
"index": "not_analyzed",
"norms": {
"enabled": true
}
},
"Year": {
"type": "string",
"boost": 5,
"index": "not_analyzed",
"norms": {
"enabled": true
},
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
},
"YearCode": {
"type": "string",
"include_in_all": false
}
}
}

And the relevant analyzers are as shown:

"analysis": {
"analyzer": {
"analyzer_startswith": {
"type": "custom",
"filter": "lowercase",
"tokenizer": "keyword"
},
"whitespace_analyzer": {
"type": "custom",
"filter": [
"lowercase",
"asciifolding"
],
"tokenizer": "whitespace"
},
"wordAnalyzer": {
"type": "custom",
"filter": [
"lowercase",
"asciifolding",
"nGram_filter"
],
"tokenizer": "whitespace"
}
},
"filter": {
"nGram_filter": {
"max_gram": "20",
"min_gram": "1",
"type": "nGram",
"token_chars": [
"letter",
"punctuation",
"symbol",
"digit"
]
}
}
}

I would like to keep the scripting to particular fields and I have used the following query:

GET ymme/ymme_type/_search
{
"query": {
"match": {
"_all":
{
"query": "2012",
"operator": "and"
}
}
},
"script_fields": {
"year_script": {
"script": "doc['Year'].value*2"
}
}
}

It is throwing the following Exception:

"error": "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[jlllRXEgTlKVYQa7rLvLNw][ymme][0]: SearchParseException[[ymme][0]: query[_all:2012],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n "query": {\n "match": {\n "_all": \n {\n "query": "2012",\n "operator": "and"\n }\n }\n },\n "script_fields": {\n "year_script": {\n "script": "doc['Year'].value2"\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for [groovy] disabled]; }{[jlllRXEgTlKVYQa7rLvLNw][ymme][1]: SearchParseException[[ymme][1]: query[_all:2012],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n "query": {\n "match": {\n "_all": \n {\n "query": "2012",\n "operator": "and"\n }\n }\n },\n "script_fields": {\n "year_script": {\n "script": "doc['Year'].value2"\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for [groovy] disabled]; }{[jlllRXEgTlKVYQa7rLvLNw][ymme][2]: SearchParseException[[ymme][2]: query[_all:2012],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n "query": {\n "match": {\n "_all": \n {\n "query": "2012",\n "operator": "and"\n }\n }\n },\n "script_fields": {\n "year_script": {\n "script": "doc['Year'].value2"\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for [groovy] disabled]; }{[jlllRXEgTlKVYQa7rLvLNw][ymme][3]: SearchParseException[[ymme][3]: query[_all:2012],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n "query": {\n "match": {\n "_all": \n {\n "query": "2012",\n "operator": "and"\n }\n }\n },\n "script_fields": {\n "year_script": {\n "script": "doc['Year'].value2"\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for [groovy] disabled]; }{[jlllRXEgTlKVYQa7rLvLNw][ymme][4]: SearchParseException[[ymme][4]: query[_all:2012],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n "query": {\n "match": {\n "_all": \n {\n "query": "2012",\n "operator": "and"\n }\n }\n },\n "script_fields": {\n "year_script": {\n "script": "doc['Year'].value*2"\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for [groovy] disabled]; }]",
"status": 400

May I know what is the reason??????, my intention is to get the response based on the scripting that I have used but not based on the default Lucene Score ..

2

Can anyone help me out when to use "script_fields" and for what purpose and how to use it with an example...

3

Hi I have used the following query:

GET /ymme/ymme_type/_search?pretty
{
"query": {
"match": {
"Make": "Acura"
}
},
"script_fields": {
"test1": {
"script": "_source.Make*2"
}
}
}

Even this query is throwing the exception, am I doing any thing wrong or do I need to set anything inorder to make use of the script_fields in search query??????

4

By default you cannot run dynamic groovy scripts (script specified inline in a request or indexed scripts) on Elasticsearch. This is for security so Elasticsearch clusters are not by default left exposed to security vunerabilities due to malicious scripts. Please read https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html#modules-scripting for more information on this as well as instructions on how use scripts without dynamic scripting (file based scripts) and how to enabled dynamic scripting.

Please read the above link in full and carefully as it directly affects the security of your cluster.

5

Hi Colin,

For that I have added the below statement in elasticsearch.yml file
script.disable_dynamic: false
So that I am able to get the response for the "script_fields" but when am trying the query..

GET /ymme/ymme_type/_search?pretty
{
"query": {
"function_score": {
"query": {
"match": {
"_all": {
"query": "2012",
"operator": "and"
}
}
},
"functions": [
{
"script_score": {
"script": "_score*doc['Year'].value"
}
}
]
}
}
}

Am getting Exception as:

"error": "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[eddpyS7lQ2Gzj0NWQr8dKw][ymme][0]: QueryPhaseExecutionException[[ymme][0]: query[filtered(function score (_all:2012,function=script[_scoredoc['Year'].value], params [null]))->cache(_type:ymme_type)],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: GroovyScriptExecutionException[MissingMethodException[No signature of method: org.elasticsearch.script.ScoreAccessor.multiply() is applicable for argument types: (java.lang.String) values: [2012]\nPossible solutions: multiply(java.lang.Character), multiply(java.lang.Number)]]; }{[eddpyS7lQ2Gzj0NWQr8dKw][ymme][1]: QueryPhaseExecutionException[[ymme][1]: query[filtered(function score (_all:2012,function=script[_scoredoc['Year'].value], params [null]))->cache(_type:ymme_type)],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: GroovyScriptExecutionException[MissingMethodException[No signature of method: org.elasticsearch.script.ScoreAccessor.multiply() is applicable for argument types: (java.lang.String) values: [2012]\nPossible solutions: multiply(java.lang.Character), multiply(java.lang.Number)]]; }{[eddpyS7lQ2Gzj0NWQr8dKw][ymme][2]: QueryPhaseExecutionException[[ymme][2]: query[filtered(function score (_all:2012,function=script[_scoredoc['Year'].value], params [null]))->cache(_type:ymme_type)],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: GroovyScriptExecutionException[MissingMethodException[No signature of method: org.elasticsearch.script.ScoreAccessor.multiply() is applicable for argument types: (java.lang.String) values: [2012]\nPossible solutions: multiply(java.lang.Character), multiply(java.lang.Number)]]; }{[eddpyS7lQ2Gzj0NWQr8dKw][ymme][3]: QueryPhaseExecutionException[[ymme][3]: query[filtered(function score (_all:2012,function=script[_scoredoc['Year'].value], params [null]))->cache(_type:ymme_type)],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: GroovyScriptExecutionException[MissingMethodException[No signature of method: org.elasticsearch.script.ScoreAccessor.multiply() is applicable for argument types: (java.lang.String) values: [2012]\nPossible solutions: multiply(java.lang.Character), multiply(java.lang.Number)]]; }{[eddpyS7lQ2Gzj0NWQr8dKw][ymme][4]: QueryPhaseExecutionException[[ymme][4]: query[filtered(function score (_all:2012,function=script[_score*doc['Year'].value], params [null]))->cache(_type:ymme_type)],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: GroovyScriptExecutionException[MissingMethodException[No signature of method: org.elasticsearch.script.ScoreAccessor.multiply() is applicable for argument types: (java.lang.String) values: [2012]\nPossible solutions: multiply(java.lang.Character), multiply(java.lang.Number)]]; }]",
"status": 500

Unable to understand this is the query that am using is Wrong??

6

It looks like your year field is mapped as a String instead of an integer. The exception is saying that it cannot multiple a number (_score) by a String (year)

7

Then how can I achieve dynamic scoring, and my intention is to change the scoring and the response need to come by giving more priority to Year field.

When am using the following query:

"query": {
"match": {
"_all": {
"query": "2012",
"operator": "and",
"fuzziness": 1,
"prefix_length": 1
}
}
}
The fuzziness is changing the response, and am getting the records with some other years like 1991, 1992, 1993,......
For that I have moved to make use of dynamic scoring as default scoring is not behaving like expected..

8

As I said before, you will need your year field to be indexed as a number not as a string. This will mean you will need to re-index your data into an index with the mapping set appropriately.

You may also want to look at the field value factor function score which will allow you to multiply the score by the value of year without using a script. You will still need year to be mapped as an integer for this though.

9

Hi Colin,
changed mappings for that fields as:
"Year": {
"type": "integer",
"boost": 4,
"index": "not_analyzed",
"norms": {
"enabled": true
}
},

And re-indexed the data again, this time I got the Exception as:

"error": "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[eddpyS7lQ2Gzj0NWQr8dKw][ymme1][0]: QueryPhaseExecutionException[[ymme1][0]: query[filtered(function score (_all:2012,function=org.elasticsearch.common.lucene.search.function.FieldValueFactorFunction@70240bcf))->cache(_type:ymme_type)],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: ElasticsearchException[java.lang.NumberFormatException: Invalid shift value in prefixCoded bytes (is encoded value really an INT?)]; nested: UncheckedExecutionException[java.lang.NumberFormatException: Invalid shift value in prefixCoded bytes (is encoded value really an INT?)];

Query I have used:

GET /ymme1/ymme_type/_search
{
"query": {
"function_score": {
"query": {
"match": {
"_all":
{
"query": "2012",
"operator": "and"
}
}
},
"field_value_factor": {
"field": "Year",
"factor":2
}
}
}
}

10

And one more thing like the "field_value_factor" which uses Year field, will this boost the scoring for the Year field as my query will have a combination of different fields, the search will took place in 4 fields like Year, Make ,Model , Engine but priority should be given to the Year field then to Make, Model and Engine..

11

Hi Is there any solution for the requirement that I have mentioned?

12

Change your mapping for Year to be:

"Year": {
"type": "integer",
"boost": 4
}
}

The other settings don't apply to non-string types.

The field_value_factor will boost the score of all documents which match the query by the factor multiplied by the value of 'Year'. This boost will happen regardless of which field is actually matched in the query. More information is availbale here: Function score query | Elasticsearch Guide [8.11] | Elastic

13

But how can I make my fuzziness to work as expected?

Am using the query :
"query": {
"match": {
"_all": {
"query": "2012",
"operator": "and"
}
}
}

My response looks like this:

2012 AM GENERAL-VPG MV-1 V8-281 4.6L SOHC
2012 CHEVROLET CAMARO All Engine
2012 CHEVROLET CAMARO v6-3564 3.6L DOHC
2012 CHEVROLET CAMARO V8-376 6.2L
2012 LAMBORGHINI AVENTADOR 12-654 6.5L DOHC
2012 LAMBORGHINI GALLARDO 10-520 DOHC

But when am adding fuzziness to my query am getting response as:(example data)

"query": {
"match": {
"_all": {
"query": "2012",
"operator": "and"
"fuzziness": 1,
"prefix_length": 1
}
}
}
Response:

1990 AM GENERAL-VPG MV-1 V8-281 4.6L SOHC
1991 CHEVROLET CAMARO All Engine
1994 CHEVROLET CAMARO v6-3564 3.6L DOHC
1997 CHEVROLET CAMARO V8-376 6.2L
2000 LAMBORGHINI AVENTADOR 12-654 6.5L DOHC
2011 LAMBORGHINI GALLARDO 10-520 DOHC

And even I tried by keeping prefix_length as 4 but even for that few responses are not matching.

I dont know why am getting this irrelevant data? May be score based response is making like this I have thought, so for that I have chosen this dynamic scoring.
May I know A better solution for my issue???

14

Hi Colin,

Thanks for your response and I found a good solution for to preserve my fuzziness and the year data preference with a new query that I have implemented as shown:

GET ymme/ymme_type/_search
{
"size": 10,
"query": {
"bool": {
"must": [
{
"match": {
"_all": {
"query": "aston,, ma",
"operator": "and",
"fuzziness": 2,
"prefix_length": 1
}
}
},
{
"query_string": {
"fields": [
"Year^4","ShortYear^5","Model^3","Engine^2"
],
"query": "70 AND 95"
}
}

     ]
  }

}
}
By this I can able to apply fuzziness and even my intention to give priority to year field is working fine..
Thanks for your support...