Painless Scripting Compile Errors

Elastic Stack Elasticsearch
1

I'm writing the following and saving it as a scripted field in elasticsearch

for(name in parsed.FIRST_NAME){
    if('Steve' in parsed.FIRST_NAME){
        return parsed.FIRST_NAME
    }
}

This givse me the error:

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"script_exception","reason":"compile error","script_stack":["... ST_NAME){\n\n    if('Chan' in parsed.FIRST_NAME){\n   ...","                             ^---- HERE"],"script":"for(name in parsed.FIRST_NAME){\n\n    if('Chan' in parsed.FIRST_NAME){\n        return parsed.FIRST_NAME\n    }\n}","lang":"painless"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"somelistcopy","node":"L0V2Tkf7SoeSWwtzU_DK-A","reason":{"type":"script_exception","reason":"compile error","script_stack":["... ST_NAME){\n\n    if('Chan' in parsed.FIRST_NAME){\n   ...","                             ^---- HERE"],"script":"for(name in parsed.FIRST_NAME){\n\n    if('Chan' in parsed.FIRST_NAME){\n        return parsed.FIRST_NAME\n    }\n}","lang":"painless","caused_by":    {"type":"illegal_argument_exception","reason":"unexpected token ['in'] was expecting one of [')']."}}}]},"status":500}
        at http://localhost:5601/bundles/kibana.bundle.js?v=16108:29:9038
        at Function.Promise.try (http://localhost:5601/bundles/commons.bundle.js?v=16108:86:22184)
        at http://localhost:5601/bundles/commons.bundle.js?v=16108:86:21568
        at Array.map (<anonymous>)
        at Function.Promise.map (http://localhost:5601/bundles/commons.bundle.js?v=16108:86:21523)
        at callResponseHandlers (http://localhost:5601/bundles/kibana.bundle.js?v=16108:29:8654)
        at http://localhost:5601/bundles/kibana.bundle.js?v=16108:28:29176
        at processQueue (http://localhost:5601/bundles/commons.bundle.js?v=16108:39:9912)
        at http://localhost:5601/bundles/commons.bundle.js?v=16108:39:10805
        at Scope.$digest (http://localhost:5601/bundles/commons.bundle.js?v=16108:39:21741)

I then altered my code to the following (note that 'parsed.FIRST_NAME' is the name of the field stored in elasticsearch):

for(int i = 0; i < doc['parsed.FIRST_NAME'].length; ++i){

    if(parsed.FIRST_NAME[i] == 'Steve'){
        return parsed.FIRST_NAME
    }
}

and am now getting this error:

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"script_exception","reason":"compile error","script_stack":["... '].length; ++i){\n\n    if(parsed.FIRST_NAME[i] == ' ...","                             ^---- HERE"],"script":"for(int i = 0; i < doc['parsed.FIRST_NAME'].length; ++i){\n\n    if(parsed.FIRST_NAME[i] == 'Chan'){\n        return parsed.FIRST_NAME\n    }\n}","lang":"painless"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"somelistcopy","node":"L0V2Tkf7SoeSWwtzU_DK-A","reason":{"type":"script_exception","reason":"compile error","script_stack":["... '].length; ++i){\n\n    if(parsed.FIRST_NAME[i] == ' ...","                             ^---- HERE"],"script":"for(int i = 0; i < doc['parsed.FIRST_NAME'].length; ++i){\n\n    if(parsed.FIRST_NAME[i] == 'Chan'){\n        return parsed.FIRST_NAME\n    }\n}","lang":"painless","caused_by":{"type":"illegal_argument_exception","reason":"Variable [parsed] is not defined."}}}]},"status":500}
    at http://localhost:5601/bundles/kibana.bundle.js?v=16108:29:9038
    at Function.Promise.try (http://localhost:5601/bundles/commons.bundle.js?v=16108:86:22184)
    at http://localhost:5601/bundles/commons.bundle.js?v=16108:86:21568
    at Array.map (<anonymous>)
    at Function.Promise.map (http://localhost:5601/bundles/commons.bundle.js?v=16108:86:21523)
    at callResponseHandlers (http://localhost:5601/bundles/kibana.bundle.js?v=16108:29:8654)
    at http://localhost:5601/bundles/kibana.bundle.js?v=16108:28:29176
    at processQueue (http://localhost:5601/bundles/commons.bundle.js?v=16108:39:9912)
    at http://localhost:5601/bundles/commons.bundle.js?v=16108:39:10805
    at Scope.$digest (http://localhost:5601/bundles/commons.bundle.js?v=16108:39:21741)

How do I get this script working? I'm not sure what exactly I'm missing here

2

Accessing fields within a scripted field is done with the special doc variable, but you need to continue to use it throughout your script. Also, I don't remember the "in" keyword, but you can use a for-each loop with :. Finally, you need to use type names, even if that type name is the dynamic def. For example:

for (String name : doc['parsed.FIRST_NAME']) {
    if ('Steve' == name){
        return name;
    }
}
return 'something else';
3

Thanks for that, I've rebuilt my code a bit :slight_smile: ! Unfortunately I have another issue now. In my index I've pushed data from two different datasets (into one index, as you can't do comparisons across indexes using elasticsearch). The fields from the dataset that are pushed in first become 'missing fields' and when I'm scripting, I'm unable to access them even though I know they're there.

When I create an index that just contains data from one dataset, I can access the FIRST_NAME field like so:

for (String name: params['_source']['FIRST_NAME']){
    if (name != null) {
        return name
    }
    
}

however once I add in the additional data, I get the following error:
Courier Fetch: 1 of 5 shards failed.

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"script_exception","reason":"runtime error","script_stack":["for (String name: params['_source']['FIRST_NAME']){\n    ","                                   ^---- HERE"],"script":"for (String name: params['_source']['FIRST_NAME']){\n    if (name != null) {\n        return name\n    }\n    \n}","lang":"painless"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"xmlxpathcopy","node":"UGEXdG44R0WjDyGbzrbZTg","reason":{"type":"script_exception","reason":"runtime error","script_stack":["for (String name: params['_source']['FIRST_NAME']){\n    ","                                   ^---- HERE"],"script":"for (String name: params['_source']['FIRST_NAME']){\n    if (name != null) {\n        return name\n    }\n    \n}","lang":"painless","caused_by":{"type":"null_pointer_exception","reason":null}}}]},"status":500}
    at http://localhost:5601/bundles/kibana.bundle.js?v=16108:29:9038
    at Function.Promise.try (http://localhost:5601/bundles/commons.bundle.js?v=16108:86:22184)
    at http://localhost:5601/bundles/commons.bundle.js?v=16108:86:21568
    at Array.map (<anonymous>)
    at Function.Promise.map (http://localhost:5601/bundles/commons.bundle.js?v=16108:86:21523)
    at callResponseHandlers (http://localhost:5601/bundles/kibana.bundle.js?v=16108:29:8654)
    at http://localhost:5601/bundles/kibana.bundle.js?v=16108:28:29176
    at processQueue (http://localhost:5601/bundles/commons.bundle.js?v=16108:39:9912)
    at http://localhost:5601/bundles/commons.bundle.js?v=16108:39:10805
    at Scope.$digest (http://localhost:5601/bundles/commons.bundle.js?v=16108:39:21741)

Is there a way to access fields that claim to be 'missing' in elasticsearch?
As an extra, when I check the mapping of the index containing both datasets, I can see the field 'FIRST_NAME'.

4

First, I highly recommend you do not use _source within a script. Depending on the type of script used (for example, a scoring script), this can be extremly costly in I/O. With that said, whether you access doc values with doc or original values with _source, you need to check if the field actually exists for the document the script is currently operating on. The field existing in mappings only means elasticsearch knows how to parse the field when indexing (among other things). But if a single document doesn't contain that field (FIRST_NAME in your case), then doc['FIRST_NAME'].value will return null, as would params._source['FIRST_NAME'].

5

Okay I'll change that now :slight_smile:
As for the field existing, I can search for, say, values contained within the 'FIRST_NAME' like 'Steve', within the specified index (using the discover tab on Kibana) and I can find them. Not all documents contain this parameter, only about 1/10th of them do however they are there within the index - they just aren't being captured by the script if that makes sense.

6

Being able to search in a field only means the mapping exists for the field. If a particular document does not have a value for the field, you cannot use doc['FIRST_NAME'].value as a blanket statement. You need to qualify this by checking if FIRST_NAME exists in doc like:

if (doc.containsKey('FIRST_NAME')) {
  return doc['FIRST_NAME'].value;
}
return 'some default value';
1 Like
7

I have come across the same error trying to use painless scripting on string data. Thanks for the help so far.

That's what my script currently looks like :

{
"script": {
"lang": "painless",
"inline": "if (doc.containsKey('request')){return doc['request'];} else return -1;"
}
}

And I still get the error : {"type":"index_out_of_bounds_exception","reason":null}}}]},"status":500}

When I replace the line by "if (doc.containsKey('request')){return 'test';} else return -1;" on the same data sample, it works and returns 100% 'test'.

How can I safely access doc['request'].value without this index_out_of_bounds_exception ?

1 Like
8

The code you pasted above returns doc['request'], not doc['request'].value. Have you tried that latter? Also, do you see any errors in your elasticsearch log?

9

Yes I have tried doc['request'].value, the above code is just a simplification in order to keep only the error part.

I do see an error in my elasticsearch log, and a big one (~200 lines).
The following error happened with the code from my previous post :

[2018-03-21T17:25:52,565][DEBUG][o.e.a.s.TransportSearchAction] [Mona01] [prodapacheflorence-2018.02.28][0], node[r8Zt9kl_QzGACqU2DBs9Ow], [R], s[ST ARTED], a[id=GmLJXFqtRSqCwb-CV1Mixw]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[prodapacheflorence-2018.02.27, prodapac heflorence-2018.02.28], indicesOptions=IndicesOptions[id=39, ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wild cards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=, routing='null', preference='1521643103178', requ estCache=null, scroll=null, source={
...
"aggregations" : {
"5" : {
"terms" : {
"field" : "request",
"script" : {
"inline" : "if (doc.containsKey('request')){return doc['request'];} else return -1;",
"lang" : "painless"
},
"size" : 5,
"min_doc_count" : 1,
"shard_min_doc_count" : 0,
"show_term_doc_count_error" : false,
"order" : [
{
"_count" : "desc"
},
{
"_term" : "asc"
...
org.elasticsearch.transport.RemoteTransportException: [Mona01][10.224.4.13:9300][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:414) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:108) ~[elasticsearch-5.5.1.jar:5.5.1]
...
Caused by: org.elasticsearch.script.ScriptException: runtime error
at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:101) ~[?:?]
...
Caused by: java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:546) ~[?:1.8.0_161]
at java.nio.DirectByteBuffer.getShort(DirectByteBuffer.java:594) ~[?:1.8.0_161]
...

I have some difficulties to understand how it can get through the condition doc.containsKey('request') and still have null error when evaluating doc['request']

10

Can you please paste a link to the entire error log message (all stack frames)? Do not remove any lines.

11

Also, if you are pasting here instead of in a gist or something like that, please put it in triple backticks (this will make it monospaced). Eg:

Caused by: whatever.Exception
12

Thanks, your suggestion to me worked and I can now access the variable and use them :slight_smile:
I have a smaller question now, with painless is it possible to use the 'in' operator?
In my case I've made two arrays (take values from the fields present in elasticsearch) called name1 and name2 and I'm trying to check if any values from name1 are present in name2
I've tried the following:

for (i:name1){
    for(j:name2){
        if (name1[i] =~ name2) {
            return 'Contact ' + name1[i] + ' found';
        }
        else {
            return 'no';
        }
    }
}

Which produces this:

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"script_exception","reason":"compile error","script_stack":["...   return 'nay';\n}\n\nfor (i:name1){\n    for(j:name2) ...","                             ^---- HERE"],"script":"def name1 = [];\ndef name2 = [];\n\nif (doc['FIRST_NAME'].value != null){\n    name1.add(doc['FIRST_NAME'].value);\n}\nif (doc['fullname'].value != null){\n    name2.add(doc['fullname'].value);\n}\n\nelse {\n    return 'nay';\n}\n\nfor (i:name1){\n    for(j:name2){\n        if (name1[i] =~ name2) {\n            return 'Contact ' + name1[i] + ' found';\n        }\n        else {\n            return 'no';\n        }\n    }\n}","lang":"painless"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"xmlxpathcopy","node":"UGEXdG44R0WjDyGbzrbZTg","reason":{"type":"script_exception","reason":"compile error","script_stack":["...   return 'nay';\n}\n\nfor (i:name1){\n    for(j:name2) ...","                             ^---- HERE"],"script":"def name1 = [];\ndef name2 = [];\n\nif (doc['FIRST_NAME'].value != null){\n    name1.add(doc['FIRST_NAME'].value);\n}\nif (doc['fullname'].value != null){\n    name2.add(doc['fullname'].value);\n}\n\nelse {\n    return 'nay';\n}\n\nfor (i:name1){\n    for(j:name2){\n        if (name1[i] =~ name2) {\n            return 'Contact ' + name1[i] + ' found';\n        }\n        else {\n            return 'no';\n        }\n    }\n}","lang":"painless","caused_by":{"type":"illegal_argument_exception","reason":"invalid sequence of tokens near [':'].","caused_by":{"type":"no_viable_alt_exception","reason":null}}}}]},"status":500}
    at http://localhost:5601/bundles/kibana.bundle.js?v=16108:29:9038
    at Function.Promise.try (http://localhost:5601/bundles/commons.bundle.js?v=16108:86:22184)
    at http://localhost:5601/bundles/commons.bundle.js?v=16108:86:21568
    at Array.map (<anonymous>)
    at Function.Promise.map (http://localhost:5601/bundles/commons.bundle.js?v=16108:86:21523)
    at callResponseHandlers (http://localhost:5601/bundles/kibana.bundle.js?v=16108:29:8654)
    at http://localhost:5601/bundles/kibana.bundle.js?v=16108:28:29176
    at processQueue (http://localhost:5601/bundles/commons.bundle.js?v=16108:39:9912)
    at http://localhost:5601/bundles/commons.bundle.js?v=16108:39:10805
    at Scope.$digest (http://localhost:5601/bundles/commons.bundle.js?v=16108:39:21741)

I've also tried this:

for (int i = 0; i < name1.length; ++i){
    if (name1[i] in name2) {
        return 'Contact ' + name1[i] + ' found';
    }
}

which returns:

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"script_exception","reason":"compile error","script_stack":["... ; ++i){\n    if (name1[i] in name2) {\n        retur ...","                             ^---- HERE"],"script":"def name1 = [];\ndef name2 = [];\n\nif (doc['FIRST_NAME'].value != null){\n    name1.add(doc['FIRST_NAME'].value);\n}\nif (doc['fullname'].value != null){\n    name2.add(doc['fullname'].value);\n}\n\nelse {\n    return 'nay';\n}\n\nfor (int i = 0; i < name1.length; ++i){\n    if (name1[i] in name2) {\n        return 'Contact ' + name1[i] + ' found';\n    }\n}","lang":"painless"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"xmlxpathcopy","node":"UGEXdG44R0WjDyGbzrbZTg","reason":{"type":"script_exception","reason":"compile error","script_stack":["... ; ++i){\n    if (name1[i] in name2) {\n        retur ...","                             ^---- HERE"],"script":"def name1 = [];\ndef name2 = [];\n\nif (doc['FIRST_NAME'].value != null){\n    name1.add(doc['FIRST_NAME'].value);\n}\nif (doc['fullname'].value != null){\n    name2.add(doc['fullname'].value);\n}\n\nelse {\n    return 'nay';\n}\n\nfor (int i = 0; i < name1.length; ++i){\n    if (name1[i] in name2) {\n        return 'Contact ' + name1[i] + ' found';\n    }\n}","lang":"painless","caused_by":{"type":"illegal_argument_exception","reason":"unexpected token ['in'] was expecting one of [')']."}}}]},"status":500}
    at http://localhost:5601/bundles/kibana.bundle.js?v=16108:29:9038
    at Function.Promise.try (http://localhost:5601/bundles/commons.bundle.js?v=16108:86:22184)
    at http://localhost:5601/bundles/commons.bundle.js?v=16108:86:21568
    at Array.map (<anonymous>)
    at Function.Promise.map (http://localhost:5601/bundles/commons.bundle.js?v=16108:86:21523)
    at callResponseHandlers (http://localhost:5601/bundles/kibana.bundle.js?v=16108:29:8654)
    at http://localhost:5601/bundles/kibana.bundle.js?v=16108:28:29176
    at processQueue (http://localhost:5601/bundles/commons.bundle.js?v=16108:39:9912)
    at http://localhost:5601/bundles/commons.bundle.js?v=16108:39:10805
    at Scope.$digest (http://localhost:5601/bundles/commons.bundle.js?v=16108:39:21741)

I've had a look online and it seems like this is a valid operator, which is leading me to believe my syntax is wrong

13

I think there are 2 features being used not quite correctly in your examples. First, a for-each loop must have a typed lefthand side (even if that type is def) and the value is set to each value of the collection, not the index of the value. Second, there is no in operator, but Java's standard Collection interface provides a contains method for this purpose. I think your example would look something like this:

for (String x : name1){
    if (name2.contains(x)) {
        return 'Contact ' + x + ' found';
    } else {
        return 'no';
    }
}

Something to keep in mind: the runtime of this loop will be quadratic on the number of names. If these are anything more than trivial lists (for example, more than a handful of elements), you should look at using Sets to do efficient intersection.

14

Ok, here it is : Full log message

Hope it helps

15

The type of script you are using (on a terms aggregation) has a slightly different syntax. The doc variable is not available. Instead, use the special _value variable, which will have the value of that field for the document the script is being run on. Also, a terms aggregation should return the type of data that the term contains, so returning -1 won't work. Instead, return some String value.

2 Likes
16

Thanks, that's exactly what I wanted.

If it can help anyone else, that's the final script :

"inline": "if (_value.length() > 12) {return _value.substring(0,12);} else return _value;"

Maybe that _value is worth being mentionned in the documentation.

Thanks

17

It has an example in the docs, albeit with little to no context.

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.