Runtime field causes the error "A document doesn't have a field"

Elastic Stack Elasticsearch
1

I'm trying to use a run time field but I keep getting the error:

A document doesn't have a value for a field! Use doc[<field>].size()==0 to check if a document is missing a field!

You can reproduce the issue by running this script

PUT player
{
  "mappings": {
    "properties": {
      "name": {
        "type": "keyword"
      },
      "create_date": {
        "type": "date",
        "format": "yyyy-MM-dd"
      },
      "achievements": {
        "type": "nested",
        "properties": {
          "quest": {
            "type": "keyword"
          },
          "points": {
            "type": "long"
          },
          "completion_date": {
            "type": "date",
            "format": "yyyy-MM-dd"
          }
        }
      }
    }
  }
}
POST player/_doc
{
  "name": "Billy Dragon",
  "create_date": "2023-01-01",
  "achievements": [
    {
      "quest":"Streets of Desolation",
      "points": 20,
      "completion_date": "2023-02-01"
    }
  ]
}
GET player/_search
{
  "runtime_mappings": {
    "new_field": {
      "type": "long",
      "script": {
        "source": """
          long current = (new Date()).getTime();
          long dueDate = doc['achievements.completion_date'].value.getMillis();
          long days = Math.round((current - dueDate)/(1000*3600*24));
          emit(days);
          """
      }
    }
  },
  "aggs": {
    "new_field": {
      "terms": {
        "field": "new_field"
      }
    }
  }
}

All this script does is create a new index, then insert one record, then do a search with run time field. Once you do the search, you will see the error I mentioned.

What did I do wrong?

2

Hi @learningelastic

"type": "nested",

Nested Field and runtime are much harder not directly supported by painless you have to loop through them etc with a much more complex script... is that what you really want?
Are you going to have an array of achievements?
If so which one do you want to access in the runtime field? All of them? The First One? The Last One?
I know this is an example but what are you really trying to accomplish?

Say the document looks like

POST player/_doc
{
  "name": "Billy Dragon",
  "create_date": "2023-01-01",
  "achievements": [
    {
      "quest": "Streets of Desolation",
      "points": 20,
      "completion_date": "2023-02-01"
    },
    {
      "quest": "Streets of Fire",
      "points": 35,
      "completion_date": "2023-01-02"
    }
  ]
}

What are you trying to do?

3

Thanks for reply Stephen!

One of the queries I wanted to run and visualize in Kibana is something like:

Graph Type: Horizontal Bar
Y Axis : Number of Days since Achievement.completion_date
X Axis: Concat(player.name, achievement[N].quest)
Filter: WHERE player.create_date >= 2023-01 && player.create_date < 2023-02-01

So the part I'm trying to figure out is how to loop through every achievement and subtract its completion_date from (new Date()).getTime().

Also, a bit later, I might want to apply a "compound interest" formula to the achievement.points , then plot y-axis as "compound interest points" and x axis is concat(player.name, achievement[x].quest)

Do you have suggestion on what I need to study? or any code examples that I can learn from?

4

In short, I would not use nested type. Nested type "looks good" and solves some problems but creates others... I would denormalize the data.

Are you going to be updating this data? ... adding achievements etc?

Yeah perfect example ... very hard / normal full programming a simple runtime field will not work.

Simple Way is This

POST player/_doc
{
  "name": "Billy Dragon",
  "create_date": "2023-01-01",
  "achievements": [
    {
      "quest": "Streets of Desolation",
      "points": 20,
      "completion_date": "2023-02-01"
    },
    {
      "quest": "Streets of Fire",
      "points": 35,
      "completion_date": "2023-01-02"
    }
  ]
}

Becomes this

POST player/_doc
{
  "name": "Billy Dragon",
  "create_date": "2023-01-01",
  "achievements": 
    {
      "quest": "Streets of Desolation",
      "points": 20,
      "completion_date": "2023-02-01"
    }
}

POST player/_doc
{
  "name": "Billy Dragon",
  "create_date": "2023-01-01",
  "achievements": 
   {
      "quest": "Streets of Fire",
      "points": 35,
      "completion_date": "2023-01-02"
    }
}

Then everything will be much easier

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.