# Null date field is epoch in painless

**URL:** <https://discuss.elastic.co/t/null-date-field-is-epoch-in-painless/138832>\
**Category:** Elasticsearch\
**Created:** [July 6, 2018, 6:40am UTC](https://discuss.elastic.co/t/null-date-field-is-epoch-in-painless/138832 "2018-07-06T06:40:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![fredericogalvao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fredericogalvao/32/33093_2.png) [@fredericogalvao](https://discuss.elastic.co/u/fredericogalvao)\
**Post date:** [July 6, 2018, 6:40am UTC](https://discuss.elastic.co/t/null-date-field-is-epoch-in-painless/138832/1 "2018-07-06T06:40:06Z")

</div>

I have an index with a mapping with a field like this:

```json
"some_date_field": {
	"type": "date"
}

```

and when that field has a value of `null`, through a **painless** script it comes as a `DateTime` representing the epoch (and some other strange behaviour I didn't know before), which just confused me for a couple of hours.

The following query:

```json
{
  "size": 1,
  "query": {
    "constant_score": {
      "filter": {
        "bool": {
          "must_not": [
            {
              "exists": {
                "field": "some_date_field"
              }
            }
          ]
        }
      }
    }
  },
  "_source": [
    "some_date_field"
  ],
  "script_fields": {
    "s": {
      "script": "params._source['some_date_field']"
    },
    "d.f.values": {
      "script": "doc['some_date_field'].values"
    },
    "d.f.values.string": {
      "script": "doc['some_date_field'].values.toString()"
    },
    "d.f.value": {
      "script": "doc['some_date_field'].value"
    },
    "d.f.value.string": {
      "script": "doc['some_date_field'].value.toString()"
    }
  }
}

```

gives me the following response:

```json
{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 6763,
    "max_score": 1,
    "hits": [
      {
        "_index": "XXX",
        "_type": "XXX",
        "_id": "XXX",
        "_score": 1,
        "_source": {
          "some_date_field": null
        },
        "fields": {
          "s": [
            null
          ],
          "d.f.values.string": [
            "[]"
          ],
          "d.f.value": [
            "1970-01-01T00:00:00.000Z"
          ],
          "d.f.value.string": [
            "1970-01-01T00:00:00.000Z"
          ]
        }
      }
    ]
  }
}

```

### Notice that:

- [expected] the source has `null`, everywhere
- [unexpected] `d.f.values` is not on the response (empty lists are not a valid value?)
- [unexpected] `d.f.values.string` shows that `d.f.values` is an empty list
- [unexpected] `d.f.value` comes as the string of the epoch DateTime
- [unexpected] `d.f.value.string` comes as the string of the epoch DateTime, showing that it was not a post-processing transforming `null` into that value (it **is** that value inside the script context)

This is part of a migration from 2.4.6 to 6.3.0, which made me realize that a `null` date field doesn't have the same behaviour it used to have on groovy scripts when on boolean contexts.

I see/know though (I've just tested on my old cluster) that in groovy/es@2.4, date fields used to be represented as `0` whenever `d.f.value` was obtained, which **is the millis equivalent to the epoch** , making the new behaviour **kinda expected** with regards to compatibility. However, neither ES versions have anything on the docs mentioning that a `null` value on date fields will be handled differently (the `null_value` section on the date datatype says it will be handled as `null` || missing), so I'm willing to say that **this has always been confusing or wrong**.

If this is by design, I'd love to be lectured on why, and the docs could gain some extra words to help clarify this.

---

<div class="post-metadata">

**Author:** ![rjernst](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rjernst/32/6363_2.png) [@rjernst](https://discuss.elastic.co/u/rjernst)\
**Post date:** [July 13, 2018, 6:03pm UTC](https://discuss.elastic.co/t/null-date-field-is-epoch-in-painless/138832/2 "2018-07-13T18:03:56Z")

</div>

`null` cannot be indexed, since it means a value does not exist. The behavior you see is because of how scripts currently expose doc values, but [this is changing](https://github.com/elastic/elasticsearch/pull/30975) to throw an error instead of returning a "default" value (epoch for date fields as you have seen). You can check if a field has any values by looking at the size of the object returned from `doc` in a script, eg `doc['myfield'].size() == 0`.

---

<div class="post-metadata">

**Author:** ![fredericogalvao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fredericogalvao/32/33093_2.png) [@fredericogalvao](https://discuss.elastic.co/u/fredericogalvao)\
**Post date:** [July 13, 2018, 7:05pm UTC](https://discuss.elastic.co/t/null-date-field-is-epoch-in-painless/138832/3 "2018-07-13T19:05:45Z")

</div>

> [@rjernst](#):
>
> doc['myfield'].size() == 0

Nice, that snippet is the suggestion I was looking for but couldn't find on the Painless API. I really think it should be noted on the docs as much as possible.

Nice to see that at least it'll throw an error instead of returning a fake value.

Thanks for your time!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 10, 2018, 7:05pm UTC](https://discuss.elastic.co/t/null-date-field-is-epoch-in-painless/138832/4 "2018-08-10T19:05:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.