All of my documents have specific field but I get "A document doesn't have a value for a field! Use doc[<field>].size()==0 to check if a document is missing a field!"

[msln]

when I run following query:

GET /annotations/_search
{
  "query": {
    "match_all": {}
  }
}

I receive following response:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "annotations",
        "_type" : "_doc",
        "_id" : "I9nlA",
        "_score" : 1.0,
        "_source" : {
          "preferences" : {
            "id" : 1,
            "annotation_id" : "I9nlA",
            "answer_timer" : 24
          }
         // other fields here
        }
      },
      {
        "_index" : "annotations",
        "_type" : "_doc",
        "_id" : "XIP6L",
        "_score" : 1.0,
        "_source" : {
          "id" : "XIP6L",
          "preferences" : {
            "id" : 2,
            "annotation_id" : "XIP6L",
            "answer_timer" : 5
          },
        // other fields here
        }
      }
    ]
  }
}

As you can see, the field Preferences has completely full without any null values.
My problem is that when I add scripted field based on the Preferences field, it raises and error.
The query:

GET /annotations/_search
{
  "query": {
    "match_all": {}
  },
  "script_fields": {
    "can_answer": {
      "script": {
        "source": """
          if (doc['creator_id'].value == params['user_id']){
            return true;
          }
          else{
            String nowString = params['now'];
            ZonedDateTime now = ZonedDateTime.parse(nowString);
            ZonedDateTime created = doc['created'].value;
            ZonedDateTime createdPlusAnswerTimer = created.plusHours(
              doc['preferences.answer_timer'].value
            );
            Duration d = Duration.between(now, createdPlusAnswerTimer);
            return d.toHours() > 0;
            }
        """,
        "params": {
          "user_id": 1,
          "now": "2022-05-17T16:17:49.366Z"
        }
      }
    }
  }
}

The error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "script_exception",
        "reason" : "runtime error",
        "script_stack" : [
          "org.elasticsearch.index.fielddata.ScriptDocValues.throwIfEmpty(ScriptDocValues.java:73)",
          "org.elasticsearch.index.fielddata.ScriptDocValues$Longs.get(ScriptDocValues.java:118)",
          "org.elasticsearch.index.fielddata.ScriptDocValues$Longs.getValue(ScriptDocValues.java:113)",
          "createdPlusAnswerTimer = created.plusHours(\n              doc['preferences.answer_timer'].value\n            );\n            Duration ",
          "                                                                                         ^---- HERE"
        ],
        "script" : " ...",
        "lang" : "painless",
        "position" : {
          "offset" : 398,
          "start" : 309,
          "end" : 441
        }
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "annotations",
        "node" : "_ljs4uLdR7eBjL8ioYLqAg",
        "reason" : {
          "type" : "script_exception",
          "reason" : "runtime error",
          "script_stack" : [
            "org.elasticsearch.index.fielddata.ScriptDocValues.throwIfEmpty(ScriptDocValues.java:73)",
            "org.elasticsearch.index.fielddata.ScriptDocValues$Longs.get(ScriptDocValues.java:118)",
            "org.elasticsearch.index.fielddata.ScriptDocValues$Longs.getValue(ScriptDocValues.java:113)",
            "createdPlusAnswerTimer = created.plusHours(\n              doc['preferences.answer_timer'].value\n            );\n            Duration ",
            "                                                                                         ^---- HERE"
          ],
          "script" : " ...",
          "lang" : "painless",
          "position" : {
            "offset" : 398,
            "start" : 309,
            "end" : 441
          },
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "A document doesn't have a value for a field! Use doc[<field>].size()==0 to check if a document is missing a field!"
          }
        }
      }
    ]
  },
  "status" : 400
}

It said there is some problems with doc['preferences.answer_timer'].value, but I don't know why. This field has value in all documents.

What is the problem and how can I solve it?
Thanks.

[stu (Stuart Tettemer)]

Hi @msln,
You can try to discover which document doesn't have a value for a field by using Debug.explain.

            if (doc['preferences.answer_timer'].empty) {
              Debug.explain(doc['preferences.id']);
            }
            ZonedDateTime createdPlusAnswerTimer = created.plusHours(
              doc['preferences.answer_timer'].value
            );

That will throw an exception with a value you can use to track down the document triggering the failure, in this case it will reference preferences.id, "to_string" : "[2]", below indicates this is preferences.id: 5 .

    "root_cause" : [
      {
        "type" : "script_exception",
        "reason" : "runtime error",
        "painless_class" : "org.elasticsearch.index.fielddata.ScriptDocValues.Longs",
        "to_string" : "[5]",
        "java_class" : "org.elasticsearch.index.fielddata.ScriptDocValues$Longs",
        "script_stack" : [
          "Debug.explain(doc['preferences.id']);\n            }\n            ZonedDateTime ",
          "                 ^---- HERE"
        ],