Dynamic mapping of date fields present in epoch_millis format

Elastic Stack Elasticsearch
1

Hi,

I was observing how date fields can be dynamically mapped.In my use case, fields that to be mapped as date are in epoch_millis format.
I have a dynamic template set for date field as :

{
          "dates": {
            "match_mapping_type": "date",
                       "mapping": {
              "type": "date",
              "dynamic_date_formats" : "epoch_millis",
              "fields": {
                "keyword": {
                  "type":  "keyword",
                  "ignore_above": 256
                }
              },
                        "copy_to": [
                            "COMMON_FIELD"
                        ]
            }
          }
        }

  1. While indexing a new field present in epoch_millis format within double quotes, this is considered as string.

    { "date_field1":"1536517800000"      }

  2. While indexing the same value in another new field present in same format(epoch_millis) without double quotes, this is considered as an integer and throws an mapper parsing exception
    as this number is out of range of int.

    {
    "date_field2": 1536517800000
    }

How do I ensure these values are taken as date rather than string and integer? Please note, I only receive values in epoch_millis format?

2

@elastic Please provide a suggestion

3

Why not using a match field name pattern instead? https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html#match-unmatch

4

Thanks @dadoonet for the response

I did use match field pattern too. Although this didn't work in my case.

{ "date_field1":"1536517800000" } was taken as text where as { "date_field1":1536517800000 } was taken as long.

However, I can't completely depend on this match field pattern.

Does not Elasticsearch handle text present in epoch_millis as a part of dynamic mapping ?

5

Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

A full reproduction script will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.

6

Hi @dadoonet
My Bad.
match field pattern did work fine.

My mapping :

 { "mappings": {
            "_doc": {
                "dynamic_templates": [
                    {
                        "date": {
                           "match": "*_date_*",
                            "mapping": {
                                "fields": {
                                    "keyword": {
                                        "ignore_above": 256,
                                        "type": "keyword"
                                    }
                                },
                                
                                "type": "date"
                            }
                        }
                    }
                ]
            }
        }
 }

Data indexed :

{
	
	"a_date_bc":1527359400000
}

Another data

{
	
	"a_date_bcc":"1527359400000"
}

GET _mapping

 "a_date_bc": {
                        "type": "date",
                        "fields": {
                            "keyword": {
                                "type": "keyword",
                                "ignore_above": 256
                            }
                        }
                    },
                    "a_date_bcc": {
                        "type": "date",
                        "fields": {
                            "keyword": {
                                "type": "keyword",
                                "ignore_above": 256
                            }
                        }
                    }

Thanks for your suggestion @dadoonet

However sometimes my field names may not always match a particular pattern.
Also, my other fields might match this pattern and might not actually contain a date.
How do I overcome these issues?

7

If you have fields like foo that might have a value like 1527359400000 and that might be a date, well... 0 looks like a valid date to me as well.

There is not really something you can do. It can be a number or a date. Who can know?

I mean that either you add some constraints to the sender and ask him to send dates with a prefix in the field name, like dt* or a_date_ or you try to do with best effort with the risk of rejecting lot of documents. You can try this though: https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-malformed.html

8

okay! Thanks @dadoonet for quick solutions

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.