Accessing nested fields in a script (null_pointer_exception)

[heikis]

Hello.

I have a nested document and I would like to do some processing on it before indexing it.
The parent document has a "reference_ts" containing a timestamp. Then come the nested fields that each have their own timestamp in "ts" field. I would like to calculate the difference between the "reference_ts" and the nested fields' "ts" in minutes.

The problem is that I can not access the nested field's "ts" value from inside the script.
How should I be accessing the nested field? Thank you!

The nested document I use for testing:

POST coolindex/_doc/
{
  "reference_ts": "2022-05-06T14:14:14.144Z",
  "nested_entries": [
    {
      "ts": "2022-04-04T14:25:25.125Z",
      "delta_ts": ""
    },
    {
      "ts": "2022-04-03T11:26:26.126Z",
      "delta_ts": ""
    },
    {
      "ts": "2022-04-02T12:27:27.127Z",
      "delta_ts": ""
    },
    {
      "ts": "2022-04-04T13:28:28.128Z",
      "delta_ts": ""
    }
  ]
}

The imported processor in my ingest pipeline:

"processors": [
  {
    "foreach": {
      "field": "nested_entries",
      "processor": {
        "script": {
          "source": "ZonedDateTime start = ZonedDateTime.parse(ctx['reference_ts']);            ZonedDateTime end = ZonedDateTime.parse(ctx['_ingest']['_value']['ts']);ctx['_ingest']['_value']['delta_ts'] = ChronoUnit.MINUTES.between(start, end);"
        }
      }
    }
  }
]

The script's source in a more readable format:

ZonedDateTime start = ZonedDateTime.parse(ctx['reference_ts']);
ZonedDateTime end = ZonedDateTime.parse(ctx['_ingest']['_value']['ts']);
ctx['_ingest']['_value']['delta_ts'] = ChronoUnit.MINUTES.between(start, end);

The error log I get:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "script_exception",
        "reason" : "runtime error",
        "script_stack" : [
          "end = ZonedDateTime.parse(ctx['_ingest']['_value']['ts']);",
          "                             ^---- HERE"
        ],
        "script" : "ZonedDateTime start = ZonedDateTime.parse(ctx['reference_ts']);ZonedDateTime end = ZonedDateTime.parse(ctx['_ingest']['_value']['ts']);ctx['_ingest']['_value']['delta_ts'] = ChronoUnit.MINUTES.between(start, end);",
        "lang" : "painless",
        "position" : {
          "offset" : 107,
          "start" : 78,
          "end" : 136
        }
      }
    ],
    "type" : "script_exception",
    "reason" : "runtime error",
    "script_stack" : [
      "end = ZonedDateTime.parse(ctx['_ingest']['_value']['ts']);",
      "                             ^---- HERE"
    ],
    "script" : "ZonedDateTime start = ZonedDateTime.parse(ctx['reference_ts']);ZonedDateTime end = ZonedDateTime.parse(ctx['_ingest']['_value']['ts']);ctx['_ingest']['_value']['delta_ts'] = ChronoUnit.MINUTES.between(start, end);",
    "lang" : "painless",
    "position" : {
      "offset" : 107,
      "start" : 78,
      "end" : 136
    },
    "caused_by" : {
      "type" : "null_pointer_exception",
      "reason" : "cannot access method/field [normalizeIndex] from a null def reference"
    }
  },
  "status" : 400
}

[RabBit_BR (andre.coelho)]

Hi @heikis !

If you cant resolve your problem, you can try this:

PUT _ingest/pipeline/testnested
{
  "description": "...",
  "processors": [
    {
      "script": {
        "source": """
            ZonedDateTime start = ZonedDateTime.parse(ctx['reference_ts']);
            for (def pair : ctx['nested_entries']) {
              ZonedDateTime end = ZonedDateTime.parse(pair.ts);
              pair.delta_ts = ChronoUnit.MINUTES.between(start, end);
            }
          """
      }
    }
  ]
}

POST _ingest/pipeline/testnested/_simulate
{
  "docs": [
    {
      "_source": {
        "reference_ts": "2022-05-06T14:14:14.144Z",
        "nested_entries": [
          {
            "ts": "2022-04-04T14:25:25.125Z",
            "delta_ts": ""
          },
          {
            "ts": "2022-04-03T11:26:26.126Z",
            "delta_ts": ""
          },
          {
            "ts": "2022-04-02T12:27:27.127Z",
            "delta_ts": ""
          },
          {
            "ts": "2022-04-04T13:28:28.128Z",
            "delta_ts": ""
          }
        ]
      }
    }
  ]
}

[heikis]

Thank you! Works beautifully