Create new field on object using painless

Elastic Stack Elasticsearch
1

Hi I'm trying to set an object field from another using an update query but am getting an error.

Currently by doc has an event object in the root of the doc with a field: id
i.e.

{
  event { id: 1 }
}

all the documents have a duration_secs field:

i.e.

{
  event { id: 1 },
  duration_secs: 10
}

But to make existing documents more ECS compliant I want to add the duration to the event object as duration..
i.e.

{
  event { id: 1, duration: 10 },
  duration_secs: 10
}

I have used an _update_by_query before, but am getting an error.
Here is my request:

POST myindex-*/_update_by_query
{
  "query": {
   "bool": {
      "should": [
        {
          "exists": {
            "field" : "duration_secs"
          }
        },
       {
        "exists": {
          "field": "event"
        }
       }
      ]
   }
  },
  "script" : {
      "inline": "ctx._source.event.duration = ctx._source.duration_secs; ",
      "lang": "painless"
  }
}

And the error is:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "script_exception",
        "reason" : "runtime error",
        "script_stack" : [
          "ctx._source.event.duration = ctx._source.duration_secs; ",
          "                 ^---- HERE"
        ],
        "script" : "ctx._source.event.duration = ctx._source.duration_secs; ",
        "lang" : "painless",
        "position" : {
          "offset" : 17,
          "start" : 0,
          "end" : 56
        }
      }
    ],
    "type" : "script_exception",
    "reason" : "runtime error",
    "script_stack" : [
      "ctx._source.event.duration = ctx._source.duration_secs; ",
      "                 ^---- HERE"
    ],
    "script" : "ctx._source.event.duration = ctx._source.duration_secs; ",
    "lang" : "painless",
    "position" : {
      "offset" : 17,
      "start" : 0,
      "end" : 56
    },
    "caused_by" : {
      "type" : "null_pointer_exception",
      "reason" : "Cannot invoke \"Object.getClass()\" because \"callArgs[0]\" is null"
    }
  },
  "status" : 400
}

I cant find any resources on the error:
Cannot invoke "Object.getClass()" because "callArgs[0]" is null
But suspect it is a scoping issue of some sort?

2

Check to see if duration_secs has an entry (a number) in every record or there might be some that are NULL. That is what I am seeing with the null_pointer_exception error.

3

My query is removing any results that dont have a duration_secs defined. The reason for this is: not all message have the duration_secs field, it is added later.

I've double checked the values in the duration_secs field, and where it exists the value is numeric.

I think its because the detination field doesnt not exist: ctx._source.event.duration

"ctx._source.event.duration = ctx._source.duration_secs; ",
"                 ^---- HERE"

But I would have thought it would create this - when i have done the same with text fields it has.

4

Try

      "inline": "ctx._source.event_duration = ctx._source.duration_secs; ",

If that works then it just doesn't let you use the dot notation like you are trying.

5

This would create another top level field, I want to update the event object with a duration field - to be ECS compliant.

6

Understand. That is just a test to see if that is where the problem is.

Can you post your mapping? Interested in the event field.

7 
PUT test/_doc/1
{
  "event": { "id": 1 },
  "duration_secs": 10
}

POST test/_update_by_query
{
  "query": {
   "bool": {
      "should": [
        {
          "exists": {
            "field" : "duration_secs"
          }
        },
       {
        "exists": {
          "field": "event"
        }
       }
      ]
   }
  },
  "script" : {
      "inline": "ctx._source.event.duration = ctx._source.duration_secs; ",
      "lang": "painless"
  }
}

GET test/_search

I ran the above and got the below results.

    "hits" : [
      {
        "_index" : "test",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "event" : {
            "duration" : 10,
            "id" : 1
          },
          "duration_secs" : 10
        }
      }
    ]
8

The mapping for event on the index is:

"event" : {
     "properties" : {
         "id" : {
              "type" : "text"
          }            
     }
}

Do i need to update the mapping on existing indices, for this field to work?

9 
{
  "test" : {
    "mappings" : {
      "properties" : {
        "duration_secs" : {
          "type" : "long"
        },
        "event" : {
          "properties" : {
            "id" : {
              "type" : "long"
            }
          }
        }
      }
    }
  }
}

That is my mapping when I run the above. Which looks like what you are using.

Can you run my test to see if that works for you?

10

Yes your test worked fine for me...

Do you think it might be worth me trying to do this using a reindex with a pipeline?

11

Possible. The mapping looks like it should work so not sure what's going on unless the data set you are working with has some odd balls in there but you said you checked that already.

12

Yes i was able to eyeball them with:

GET myindex-*/_search
{
  "_source": ["duration_secs"],
"query": {
   "bool": {
      "should": [
        {
          "exists": {
            "field" : "duration_secs"
          }
        },
       {
        "exists": {
          "field": "event"
        }
       }
      ]
   }
  }
}

there is a few months of data but there aren't all that many docs, and far fewer have duration_secs

Thanks for your help - I've got some other work to look at on same dataset, so will try a reindex instead.

thanks again.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.