Date fields not being treated as timestamp fields in Kibana index patterns

[abhinav_chourasia]

I have been trying to get the mapping correct for my date values in my documents. I have come down to doing a lot of hit and trial now with it. Every time and no matter what option I try, I am not getting the date field in my document treated as a timestamp column while creating Kibana index patterns.

Here are the various options I have tried (hit and trial because I feel like a total noob with ELK ) with my index mappings:

trial 1:

{
    "mappings" : {
        "properties" : {
                 "Created" : {
          "type" : "date",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          },
          "format": "strict_date_optional_time||epoch_millis"
        },
        "Due date" : {
          "type" : "date",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          },
          "format": "strict_date_optional_time||epoch_millis"
        },
        "Updated" : {
          "type" : "date",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          },
          "format": "strict_date_optional_time||epoch_millis"
        }
      }
    }
}

trial 2

    {
    "mappings" : {
        "properties" : {
                 "Created" : {
          "type" : "date",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          }          
        },
        "Due date" : {
          "type" : "date",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          }
        },
        "Updated" : {
          "type" : "date",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
}

trial 3

    {
    "mappings" : {
        "properties" : {
                 "Created" : {
          "type" : "date",
          "fields" : {
            "keyword" : {
              "type" : "date"
            }
          }          
        },
        "Due date" : {
          "type" : "date",
          "fields" : {
            "keyword" : {
              "type" : "date"
            }
          }
        },
        "Updated" : {
          "type" : "date",
          "fields" : {
            "keyword" : {
              "type" : "date"
            }
          }
        }
      }
    }
}

And then each of the trials above with the below :

    "_default_": {
  "_timestamp": {
    "enabled": true,
    "store": true,
    "_field_names": "_timestamp"
  }
},

and then each of the

  "fields" : {
    "keyword" : {
      "type" : "date"
    }

specified with the format again explicitly with each of the above combinations.
and that’s how my sample document looks like (different trials)

trial doc 1

{ "Created": "15/11/21 13:21",
  "Updated": "30/12/21 14:30",
  "Due date": null
}

tried doc 2

{ 
  "Created": 1636982460000,
  "Updated": 1640874600000,
  "Due date": null
}

and none of the above combinations seems to make the needed fields as a timestamp field while creating the index patterns in Kibana, (Elasticsearch & Kibana 7.15)

I am sure there is a lot to learn here. Could someone please guide in the right direction? Lest I end up writing an automation for generating the various combinations (and ofcourse that would be brainless & for sure not lead to any good )

[tsullivan (Tim Sullivan)]

Have you tried without using multi-fields?

1. Add an index template that will match the indices with your data:

PUT /_index_template/abhinavlogs-dev
{
  "index_patterns": [ "abhinavlogs-*" ],
  "template": {
    "mappings": {
      "properties": {
        "Created": { "type": "date" },
        "Updated": { "type": "date" },
        "DueDate": { "type": "date" } # Note: do not put spaces in the field name.
      }
    }
  }
}

2. Index data into the matching index:

POST /abhinavlogs-001/_doc
{
  "Created": "2022-01-25T20:02:13Z",
  "Updated": "2022-01-25T20:02:13Z",
  "DueDate": null
}

3. Check the mapping of the index

GET /abhinavlogs-001/_mapping

I haven't tried using multi-fields for the date fields, but if you start with this, maybe you can work up to what you really need.

[tsullivan (Tim Sullivan)]

BTW, you don't need to have spaces in the field name. You can put a custom label on the field to make it show up with a space in Kibana: