What is this added "field" thing?

[EricJohnson]

Greetings. This is probably a simple question.

I'm creating records that look like this:

{
  "_index": "metrics_daily_summary",
  "_type": "_doc",
  "_id": "E36Bbm4B8gCymL1PmIcS",
  "_version": 1,
  "_score": 0,
  "_source": {
    "oss.id": "01",
    "statDate": "2019-11-14",
    "set": "system.cpu",
    "subset": "iowait.pct",
    "metrics": {
  "iowait.pct.mean": 0.0038915509259259263,
  "iowait.pct.median": 0.001,
  "iowait.pct.std": 0.03271507496555733,
  "iowait.pct.min": 0,
  "iowait.pct.max": 1.9454,
  "iowait.pct.mode": "unimodal"
    }
  },
  "fields": {
    "statDate": [
      "2019-11-14T00:00:00.000Z"
    ]
  }
}

I'm not adding this part though. What is it?

"fields": {
    "statDate": [
      "2019-11-14T00:00:00.000Z"
    ]
  }

I did PUT a index template for this index - but the "fields" thing is still there.

PUT _template/template_1
{
  "index_patterns": ["metrics_daily_summary"],
  "mappings": {
    "properties": {
      "statDate": {
        "type": "date",
        "format": "yyyy-MM-dd"
      }
    }
  }
}

Thank you for your input!

[Glen_Smith]

Note that an index_template is only applied at the time an index is created.

Take a look at the existing mapping. I think you'll find that you have "store": true on the statDate field, which results in the value for that field being store independent of the document _source.

[EricJohnson]

Hi Glen. Thank you for the reply. All it says in the mapping is:

"statDate": {
    "type": "date"
}

Any other ideas?

[Glen_Smith]

Can your provide:

• version of Elasticsearch
• query for retrieving the mapping
• full mapping response (redacted as needed)
• query for retrieving the example document
• full document response (redacted as needed)

[EricJohnson]

• version of Elasticsearch

  7.3.1

• query for retrieving the mapping

  GET /metrics_daily_summary/_mapping

• full mapping response (redacted as needed)

{
"metrics_daily_summary" : {
"mappings" : {
"properties" : {
"metrics" : {
"set" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256 ....
},
"statDate" : {
"type" : "date"
},
"subset" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}

• query for retrieving the example document

  GET metrics_daily_summary/_doc/CH6Bbm4B8gCymL1P6Ijn

• full document response (redacted as needed)

{
"_index" : "metrics_daily_summary",
"_type" : "_doc",
"_id" : "CH6Bbm4B8gCymL1P6Ijn",
"_version" : 1,
"_seq_no" : 10,
"_primary_term" : 1,
"found" : true,
"_source" : {
"oss.id" : "56781eaa7394c9297bcac6e2",
"statDate" : "2019-11-14",
"set" : "system.cpu",
"subset" : "iowait.pct",
"metrics" : {
"iowait.pct.mean" : 0.003682164351851852,
"iowait.pct.median" : 0.0,
"iowait.pct.std" : 0.04443670345111818,
"iowait.pct.min" : 0.0,
"iowait.pct.max" : 1.9434,
"iowait.pct.mode" : "unimodal"
}
}
}

If I look at that exact record in Kibana it shows up like this:

[EricJohnson]

[Glen_Smith]

Ah, in the Kibana Discover app.

Over on the upper left, from "New Save Open Share Inspect", select "Inspect".

A panel will slide out from the right. From "Statistics Request Response", you can select "Request" to see what the request looks like that Kibana is sending in Discover. In your case, there is probably a "docvalue_fields" key with

{
      "field": "statDate",
      "format": "date_time"
},

That's how it gets in your response body.

[EricJohnson]

Thanks again for your response. Yes. In the Request is

"docvalue_fields": [
    {
      "field": "statDate",
      "format": "date_time"
    }
  ]

So that means that this extra stuff is a Kibana format thing and there isn't anything extra being put in the index. I don't think that will be a problem.

It was really important to clarify what was going on here. We appreciate the help!