Kibana 7.9.1 - Date type mapping error

My data dates are in the epoch format thus I am trying to use mapping to display the date in the proper format in Kibana, however, it's failing with the following error message:

[2020-09-10T18:39:43,718][INFO ][o.e.a.b.TransportShardBulkAction] [tx-kibana] [fscd-flat-2020-09-05][0] mapping update rejected by primary
java.lang.IllegalArgumentException: mapper [UploadTimestamp] cannot be changed from type [date] to [text]

image

This error message is saying one thing, but your question is saying something different. Can you please be more specific about:

  1. What have you tried so far?
  2. What were you expecting to happen?

In particular, you need to explain whether you already have documents indexed, whether you already have an index template, and what the timestamps you have look like

In Kibana I am getting the expected data but the date field called 'UploadTimestamp' is displaying in epoch format image - the expected format is similar to this image (Kibana 6)
This is my date configuration in Kibana:



If I add the following mapping to my template, I stop receiving the data and notice those errors in the Kibana log

[2020-09-10T20:10:24,298][INFO ][o.e.a.b.TransportShardBulkAction] [tx-kibana] [fscd-flat-2020-09-08][0] mapping update rejected by primary
java.lang.IllegalArgumentException: mapper [UploadTimestamp] cannot be changed from type [date] to [text]
at org.elasticsearch.index.mapper.FieldMapper.mergeSharedOptions(FieldMapper.java:322) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:304) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:50) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.ObjectMapper.doMerge(ObjectMapper.java:529) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.RootObjectMapper.doMerge(RootObjectMapper.java:296) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.ObjectMapper.merge(ObjectMapper.java:489) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.RootObjectMapper.merge(RootObjectMapper.java:291) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.Mapping.merge(Mapping.java:99) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.DocumentMapper.merge(DocumentMapper.java:320) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:480) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:427) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:359) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:281) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:164) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:209) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:116) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:82) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:161) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:710) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.1.jar:7.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]

Date formats in Kibana are complete separate from date formats in the mappings of your Elasticsearch index. You are seeing Elasticsearch errors, not Kibana errors. The number you're showing me is not in the standard epoch_millis format, it's in epoch_seconds format. Take a look at the documentation here: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html

Thanks @wylie for your prompt responses but it seems that I am still missing something:
So I ran the following command in Kibana but did not resolve the issue:

PUT fscd-flat-2020-09-07
{
"mappings": {
"properties": {
"UploadTimestamp": {
"type": "date",
"format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ"
}
}
}
}

Results:
{
"fscd-flat-2020-09-07" : {
"mappings" : {
"dynamic" : "true",
"_source" : {
"includes" : ,
"excludes" :
},
"dynamic_date_formats" : ,
"date_detection" : true,
"numeric_detection" : true
}
}
}

I am still learning Kibana :blush:. What would it be your suggestion to resolve this issue?
Thanks in advance.

The information you've shared so far doesn't help me to understand where the issue is happening. Please share an example date that you are trying to store in Elasticsearch, and let's go from there.

date data in both JSON and Table format in Kibana 7:

"mac24": "xxxxxx",
"sw_ipport_desc": "xxxxxxxxxxx",
"sw_voip_port": "false",
**"UploadTimestamp": "1599592944165",** 
"CustVertical": "xxxxxxxx",
"sw_port_desc": "fa10",

image

This is date format I would like to see in Kibana:
UploadTimestamp September 7th 2020, 15:52:20.554

Please let me know if you need any additional info

Using this example:

  1. Your index can use the standard date mapping because it's in the format epoch_millis. Do not change the date format unless you need to.

  2. You can verify that your documents are correctly understood by Elasticsearch by asking it to print the data in another date format, like this:

POST demo-timestamps-2/_search
{
  "_source": true,
  "fields": [{
    "field": "UploadTimestamp",
    "format": "date"
  }]
}
  1. To change the format in Kibana you can either change the Kibana settings which you showed above, or:

  2. You can change it for just the field in the index pattern you care about:

Finally, it seems like you have gotten your data into a bad state by messing with this. To test this I reproduced the following scenario:

  1. Create documents with the wrong date format, as verified by the error on retrieval:
PUT demo-timestamps
PUT demo-timestamps/_mapping
{
  "properties": {
    "UploadTimestamp": {
      "type": "date",
      "format": "epoch_second"
    }
  }
}

PUT demo-timestamps/_doc/1
{
  "UploadTimestamp": 1599592944165
}

POST demo-timestamps/_search
{
  "_source": true,
  "fields": [{
    "field": "UploadTimestamp",
    "format": "date"
  }]
}
  1. To fix this issue, I reindexed to a correctly-mapped field:
PUT demo-timestamps-2
PUT demo-timestamps-2/_mapping
{
  "properties": {
    "UploadTimestamp": {
      "type": "date"
    }
  }
}

POST _reindex
{
  "source": {
    "index": "demo-timestamps"
  },
  "dest": {
    "index": "demo-timestamps-2"
  }
}
1 Like

THANKS for your suggestion. It worked for the existing index, once new index are uploaded to Kibana, I get this new problem:

That error message is pretty clear to me: one of your indices is in a bad state, so try out the reindex method I showed above to create a copy with the correct mappings

You're are correct, when I reindex it and delete the 'bad' index the issue goes away. is there a way to make it automated? The same data goes to Kibana 6 (configured previously by another team) and there is no issue there.

I think your approach of setting the index template from your first post would work once your existing indices match