Null Pointer exception when trying to reindex using script

Elastic Stack Elasticsearch
1

Dealing with a user-defined object in about 600k records and want to reindex to standardized field naming. Issue is field has space in name and I'm struggling to figure syntax in script to reference old field with space in name to move to field without.

Ideas/suggestions?

Reindex attempt

{
    "source": {
        "index": "records-test"
    },
    "dest": {
        "index": "records-reindex-20180924"
    },
    "script": {
        "source": "ctx._source.subject.desired_location.point.lat = ctx._source.subject.list.custom_fields['Desired location'].value.location.point.lat; ctx._source.subject.desired_location.point.lon = ctx._source.subject.list.custom_fields['Desired location'].value.location.point.lon; ctx._source.subject.list.custom_fields.remove('Desired location')"
    }
}

Yields the following error:

{
  "error": {
    "root_cause": [
      {
        "type": "script_exception",
        "reason": "runtime error",
        "script_stack": [
          "ctx._source.subject.desired_location.point.lat = ctx._source.subject.list.custom_fields['Desired location'].value.location.point.lat; ",
          "                                    ^---- HERE"
        ],
        "script": "ctx._source.subject.desired_location.point.lat = ctx._source.subject.list.custom_fields['Desired location'].value.location.point.lat; ctx._source.subject.desired_location.point.lon = ctx._source.subject.list.custom_fields['Desired location'].value.location.point.lon; ctx._source.subject.list.custom_fields.remove('Desired location')",
        "lang": "painless"
      }
    ],
    "type": "script_exception",
    "reason": "runtime error",
    "script_stack": [
      "ctx._source.subject.desired_location.point.lat = ctx._source.subject.list.custom_fields['Desired location'].value.location.point.lat; ",
      "                                    ^---- HERE"
    ],
    "script": "ctx._source.subject.desired_location.point.lat = ctx._source.subject.list.custom_fields['Desired location'].value.location.point.lat; ctx._source.subject.desired_location.point.lon = ctx._source.subject.list.custom_fields['Desired location'].value.location.point.lon; ctx._source.subject.list.custom_fields.remove('Desired location')",
    "lang": "painless",
    "caused_by": {
      "type": "null_pointer_exception",
      "reason": null
    }
  },
  "status": 500
}

Updated with new attempt and related error

Even trying to simplify to copy point field to point field fails with NPE

POST /_reindex
{
    "source": {
        "index": "records-test"
    },
    "dest": {
        "index": "records-reindex-20180924"
    },
    "script": {
        "source": "ctx._source.subject.desired_location.point = ctx._source.subject.list.custom_fields['Desired location'].value.location.point; ctx._source.subject.list.custom_fields.remove('Desired location')"
    }
}

Yields:

{
  "error": {
    "root_cause": [
      {
        "type": "script_exception",
        "reason": "runtime error",
        "script_stack": [
          "ctx._source.subject.desired_location.point = ctx._source.subject.list.custom_fields['Desired location'].value.location.point; ",
          "                                    ^---- HERE"
        ],
        "script": "ctx._source.subject.desired_location.point = ctx._source.subject.list.custom_fields['Desired location'].value.location.point; ctx._source.subject.list.custom_fields.remove('Desired location')",
        "lang": "painless"
      }
    ],
    "type": "script_exception",
    "reason": "runtime error",
    "script_stack": [
      "ctx._source.subject.desired_location.point = ctx._source.subject.list.custom_fields['Desired location'].value.location.point; ",
      "                                    ^---- HERE"
    ],
    "script": "ctx._source.subject.desired_location.point = ctx._source.subject.list.custom_fields['Desired location'].value.location.point; ctx._source.subject.list.custom_fields.remove('Desired location')",
    "lang": "painless",
    "caused_by": {
      "type": "null_pointer_exception",
      "reason": null
    }
  },
  "status": 500
}
4

I've tried the painless syntax new () as well thinking perhaps the field had to be initiated first.

Odd as I've reindex others that generate new fields without issue so unsure why the NPE here.

5

It appears the Null Pointer Exception may be caused if the source field doesn't exist. I added conditional logic and a noop operation in the else block and the reindex worked.

Just a generic NPE is not helpful and the error should indicate that a field could not be found, or perhaps save a default value if null. This error message implied that that sink field had issues when in fact, the source field for some records was missing.

I think this is solved (at least the mystery NPE), but the error could be more useful.

Working code

{
    "source": {
        "index": "records-test"
    },
    "dest": {
        "index": "records-reindex-20180924"
    },
    "script": {
        "source": "if( ctx._source.subject.list.custom_fields.containsKey(\"Desired location\") ){ ctx._source.subject.desired_location = ctx._source.subject.list.custom_fields['Desired location'].value } else if( ctx._source.subject.list.custom_fields.containsKey(\"Property location\") ) { ctx._source.subject.desired_location = ctx._source.subject.list.custom_fields['Property location'].value } else { ctx.op = \"noop\"; }"
    }
}
6

I appreciate the feedback here and agree error messages are often fairly obtuse in using scripts. Error message improvement is included in the long-term roadmap for Painless noted as one of the items in this issue (https://github.com/elastic/elasticsearch/issues/31009).

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.