Forcing consistent response when using _source

[mikkelduif (Mikkel)]

Hi there,
I have a question about using the "_source" field, where I have noticed that the response is not always consistent with the document, and would like to ask if there is some way to tweak the elasticsearch behaviour to return more consistent responses.

Given the following index:

PUT /test
{
  "mappings": {
    "properties": {
      "foo": {
        "properties": {
          "bar": {
            "type": "text"
          }
        }
      }
    }
  }
}

PUT /test/_doc/1
{
  "foo": {
    "bar": "baz"
  }
}

PUT /test/_doc/2
{
  "foo": null
}

Then when I apply the "_source" parameter I get different responses when I put "foo" and "foo.bar" in the includes on document with ID=2.

Doing the following query:

GET /test/_search
{
  "_source": {
    "includes": "foo"
  }
}

Gives me:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "test",
        "_id": "1",
        "_score": 1,
        "_source": {
          "foo": {
            "bar": "baz"
          }
        }
      },
      {
        "_index": "test",
        "_id": "2",
        "_score": 1,
        "_source": {
          "foo": null
        }
      }
    ]
  }
}

While the following query:

GET /test/_search
{
  "_source": {
    "includes": "foo.bar"
  }
}

Gives me:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "test",
        "_id": "1",
        "_score": 1,
        "_source": {
          "foo": {
            "bar": "baz"
          }
        }
      },
      {
        "_index": "test",
        "_id": "2",
        "_score": 1,
        "_source": {}
      }
    ]
  }
}

The specific question is about the response structure on document with ID=2, where we have the following difference:

{
    "foo": null
}

vs.

{}

I wish that in all circumstances it would return:

{
    "foo": null
}

I assume this happens because the document itself is not aware of the fact that "foo.bar" exists, so it's confused about the path to "foo.bar" and cannot understand that this is supposed to be a child key of the "foo" document - even though this is mentioned by the mappings.

So in short: is there any way to modify this behaviour of the "_source"?