What is this added "field" thing?

Elastic Stack Elasticsearch
1

Greetings. This is probably a simple question.

I'm creating records that look like this:

{
  "_index": "metrics_daily_summary",
  "_type": "_doc",
  "_id": "E36Bbm4B8gCymL1PmIcS",
  "_version": 1,
  "_score": 0,
  "_source": {
    "oss.id": "01",
    "statDate": "2019-11-14",
    "set": "system.cpu",
    "subset": "iowait.pct",
    "metrics": {
  "iowait.pct.mean": 0.0038915509259259263,
  "iowait.pct.median": 0.001,
  "iowait.pct.std": 0.03271507496555733,
  "iowait.pct.min": 0,
  "iowait.pct.max": 1.9454,
  "iowait.pct.mode": "unimodal"
    }
  },
  "fields": {
    "statDate": [
      "2019-11-14T00:00:00.000Z"
    ]
  }
}

I'm not adding this part though. What is it?

"fields": {
    "statDate": [
      "2019-11-14T00:00:00.000Z"
    ]
  }

I did PUT a index template for this index - but the "fields" thing is still there.

PUT _template/template_1
{
  "index_patterns": ["metrics_daily_summary"],
  "mappings": {
    "properties": {
      "statDate": {
        "type": "date",
        "format": "yyyy-MM-dd"
      }
    }
  }
}

Thank you for your input!

2

Note that an index_template is only applied at the time an index is created.

Take a look at the existing mapping. I think you'll find that you have "store": true on the statDate field, which results in the value for that field being store independent of the document _source.

3

Hi Glen. Thank you for the reply. All it says in the mapping is:

"statDate": {
    "type": "date"
}

Any other ideas?

4

Can your provide:

  • version of Elasticsearch
  • query for retrieving the mapping
  • full mapping response (redacted as needed)
  • query for retrieving the example document
  • full document response (redacted as needed)
5

  • version of Elasticsearch

    7.3.1

  • query for retrieving the mapping

    GET /metrics_daily_summary/_mapping

  • full mapping response (redacted as needed)

{
"metrics_daily_summary" : {
"mappings" : {
"properties" : {
"metrics" : {
"set" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256 ....
},
"statDate" : {
"type" : "date"
},
"subset" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}

  • query for retrieving the example document

    GET metrics_daily_summary/_doc/CH6Bbm4B8gCymL1P6Ijn

  • full document response (redacted as needed)

{
"_index" : "metrics_daily_summary",
"_type" : "_doc",
"_id" : "CH6Bbm4B8gCymL1P6Ijn",
"_version" : 1,
"_seq_no" : 10,
"_primary_term" : 1,
"found" : true,
"_source" : {
"oss.id" : "56781eaa7394c9297bcac6e2",
"statDate" : "2019-11-14",
"set" : "system.cpu",
"subset" : "iowait.pct",
"metrics" : {
"iowait.pct.mean" : 0.003682164351851852,
"iowait.pct.median" : 0.0,
"iowait.pct.std" : 0.04443670345111818,
"iowait.pct.min" : 0.0,
"iowait.pct.max" : 1.9434,
"iowait.pct.mode" : "unimodal"
}
}
}

If I look at that exact record in Kibana it shows up like this:

kibana
kibana898×1024 97.5 KB

6

kibana
kibana.png898×1024 104 KB

7

Ah, in the Kibana Discover app.

Over on the upper left, from "New Save Open Share Inspect", select "Inspect".

A panel will slide out from the right. From "Statistics Request Response", you can select "Request" to see what the request looks like that Kibana is sending in Discover. In your case, there is probably a "docvalue_fields" key with

{
      "field": "statDate",
      "format": "date_time"
},

That's how it gets in your response body.

8

Thanks again for your response. Yes. In the Request is

"docvalue_fields": [
    {
      "field": "statDate",
      "format": "date_time"
    }
  ]

So that means that this extra stuff is a Kibana format thing and there isn't anything extra being put in the index. I don't think that will be a problem.

It was really important to clarify what was going on here. We appreciate the help!

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.