Painless: problem accessing nested object using dot notation

Hi,

I'm having problems reindexing data using a painless script.

The nested field context.data.result is mapped as an object in the destination index, but in some cases the source data is either a string or an array of longs

The script attempts to:

  1. Check if the field exists
  2. Check the field's type
  3. If it is a string or array
    • create a temporary string / array variable
    • assign the value of the offending field to this variable
    • change the value of the field to be a new object (passed in via params) using a put command
    • assign the value of the temp variable to the first key/value pair of the new object

This works fine for strings, but it's not working for arrays.

Here's one document it fails on:

{
    "_index": "logstash-crm-2018.04.29",
    "_type": "doc",
    "_id": "ssNuEGMBPRb2-UEOO0NF",
    "_version": 1,
    "found": true,
    "_source": {
        "context": {
            "data": {
                "result": [
                    3433229,
                    3433301,
                    2993663
                ],
                "resultCount": 3
            },
            "message": "success",
            "status": 200
        },
        "message": "DOCUMENT_VALIDATIONS_NEEDED",
        "type": "log",
        "level": 200,
        "log_type": "json",
        "source": "/var/www/our-crm/log/our-crm-2018-04-29.log",
        "datetime": {
            "timezone_type": 3,
            "date": "2018-04-29 09:04:08",
            "timezone": "Europe/London"
        },
        "@version": "1",
        "level_name": "INFO",
        "tags": [
            "beats_input_codec_plain_applied"
        ],
        "@timestamp": "2018-04-29T08:04:09.437Z",
        "input_type": "log",
        "service": "crm",
        "offset": 82722,
        "beat": {
            "version": "5.6.8",
            "name": "aws-hostname",
            "hostname": "aws-hostname"
        },
        "log_source": "file"
    }
}

And here's my Painless script:

{
  "source": {
    "index": ["{{source}}"]
  },
  "dest": {
    "index": "{{dest}}"
  },
  "script": {
    "source": """
    if (ctx._source.context != null) {
        if (ctx._source.context instanceof String) {
            String name = ctx._source.context;
            ctx._source.put(\"context\", params.context);
            ctx._source.context.name = name 
        } else if (ctx._source.context.data != null) {
            if (ctx._source.context.data.result instanceof Long) {
                Long number = ctx._source.context.data.result;
                ctx._source.put(\"context.data.result\", params.result);
                ctx._source.context.data.result.number = number 
            } else if (ctx._source.context.data.result instanceof List) {
                    long[] longArray = new long[ctx._source.context.data.result.length];
                    longArray = ctx._source.context.data.result;
                    ctx._source.put(\"context.data.result\", params.result2);
                    ctx._source.context.data.result.longArray = longArray 
            } 
        } 
    }
    """,
    "lang": "painless",
     "params": {
     "context": { "name": "blank" },
     "result": { "number": 1234 },
     "result2": { "longArray": [1234, 5678] }
    }
  }
}

Currently Painless throws an error at this point in the code:

            "if (ctx._source.context.data != null) {",
            "                       ^---- HERE"

And the error message is:

            "type": "illegal_argument_exception",
            "reason": "Illegal list shortcut value [data]."

I'm not sure why this would be the case, since I know the source data contains context.data.result?

It looks like ctx._source.context is an array in your source. So you need to iterate through each value of the array and dereference it by index. So instead of ctx._source.context.data, you need a loop like:

for (int i = 0; i < ctx._source.context.length; ++i) {
  ctx._source.context[i].data
}
1 Like

Hi Ryan, thanks for your reply!

result is definitely an array but context is an object, unless I'm misunderstanding something?

Are you sure all of your documents have context as an object? The error from painless indicates it found a List.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.