Painless: problem accessing nested object using dot notation

Elastic Stack Elasticsearch
1

Hi,

I'm having problems reindexing data using a painless script.

The nested field context.data.result is mapped as an object in the destination index, but in some cases the source data is either a string or an array of longs

The script attempts to:

  1. Check if the field exists
  2. Check the field's type
  3. If it is a string or array
    • create a temporary string / array variable
    • assign the value of the offending field to this variable
    • change the value of the field to be a new object (passed in via params) using a put command
    • assign the value of the temp variable to the first key/value pair of the new object

This works fine for strings, but it's not working for arrays.

Here's one document it fails on:

{
    "_index": "logstash-crm-2018.04.29",
    "_type": "doc",
    "_id": "ssNuEGMBPRb2-UEOO0NF",
    "_version": 1,
    "found": true,
    "_source": {
        "context": {
            "data": {
                "result": [
                    3433229,
                    3433301,
                    2993663
                ],
                "resultCount": 3
            },
            "message": "success",
            "status": 200
        },
        "message": "DOCUMENT_VALIDATIONS_NEEDED",
        "type": "log",
        "level": 200,
        "log_type": "json",
        "source": "/var/www/our-crm/log/our-crm-2018-04-29.log",
        "datetime": {
            "timezone_type": 3,
            "date": "2018-04-29 09:04:08",
            "timezone": "Europe/London"
        },
        "@version": "1",
        "level_name": "INFO",
        "tags": [
            "beats_input_codec_plain_applied"
        ],
        "@timestamp": "2018-04-29T08:04:09.437Z",
        "input_type": "log",
        "service": "crm",
        "offset": 82722,
        "beat": {
            "version": "5.6.8",
            "name": "aws-hostname",
            "hostname": "aws-hostname"
        },
        "log_source": "file"
    }
}

And here's my Painless script:

{
  "source": {
    "index": ["{{source}}"]
  },
  "dest": {
    "index": "{{dest}}"
  },
  "script": {
    "source": """
    if (ctx._source.context != null) {
        if (ctx._source.context instanceof String) {
            String name = ctx._source.context;
            ctx._source.put(\"context\", params.context);
            ctx._source.context.name = name 
        } else if (ctx._source.context.data != null) {
            if (ctx._source.context.data.result instanceof Long) {
                Long number = ctx._source.context.data.result;
                ctx._source.put(\"context.data.result\", params.result);
                ctx._source.context.data.result.number = number 
            } else if (ctx._source.context.data.result instanceof List) {
                    long[] longArray = new long[ctx._source.context.data.result.length];
                    longArray = ctx._source.context.data.result;
                    ctx._source.put(\"context.data.result\", params.result2);
                    ctx._source.context.data.result.longArray = longArray 
            } 
        } 
    }
    """,
    "lang": "painless",
     "params": {
     "context": { "name": "blank" },
     "result": { "number": 1234 },
     "result2": { "longArray": [1234, 5678] }
    }
  }
}

Currently Painless throws an error at this point in the code:

            "if (ctx._source.context.data != null) {",
            "                       ^---- HERE"

And the error message is:

            "type": "illegal_argument_exception",
            "reason": "Illegal list shortcut value [data]."

I'm not sure why this would be the case, since I know the source data contains context.data.result?

2

It looks like ctx._source.context is an array in your source. So you need to iterate through each value of the array and dereference it by index. So instead of ctx._source.context.data, you need a loop like:

for (int i = 0; i < ctx._source.context.length; ++i) {
  ctx._source.context[i].data
}
1 Like
3

Hi Ryan, thanks for your reply!

result is definitely an array but context is an object, unless I'm misunderstanding something?

4

Are you sure all of your documents have context as an object? The error from painless indicates it found a List.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.