My data dates are in the epoch format thus I am trying to use mapping to display the date in the proper format in Kibana, however, it's failing with the following error message:
[2020-09-10T18:39:43,718][INFO ][o.e.a.b.TransportShardBulkAction] [tx-kibana] [fscd-flat-2020-09-05][0] mapping update rejected by primary
java.lang.IllegalArgumentException: mapper [UploadTimestamp] cannot be changed from type [date] to [text]
This error message is saying one thing, but your question is saying something different. Can you please be more specific about:
In particular, you need to explain whether you already have documents indexed, whether you already have an index template, and what the timestamps you have look like
In Kibana I am getting the expected data but the date field called 'UploadTimestamp' is displaying in epoch format 
- the expected format is similar to this 
(Kibana 6)
This is my date configuration in Kibana:
If I add the following mapping to my template, I stop receiving the data and notice those errors in the Kibana log
[2020-09-10T20:10:24,298][INFO ][o.e.a.b.TransportShardBulkAction] [tx-kibana] [fscd-flat-2020-09-08][0] mapping update rejected by primary
java.lang.IllegalArgumentException: mapper [UploadTimestamp] cannot be changed from type [date] to [text]
at org.elasticsearch.index.mapper.FieldMapper.mergeSharedOptions(FieldMapper.java:322) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:304) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:50) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.ObjectMapper.doMerge(ObjectMapper.java:529) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.RootObjectMapper.doMerge(RootObjectMapper.java:296) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.ObjectMapper.merge(ObjectMapper.java:489) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.RootObjectMapper.merge(RootObjectMapper.java:291) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.Mapping.merge(Mapping.java:99) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.DocumentMapper.merge(DocumentMapper.java:320) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:480) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:427) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:359) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:281) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:164) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:209) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:116) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:82) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:161) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:710) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.1.jar:7.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
Date formats in Kibana are complete separate from date formats in the mappings of your Elasticsearch index. You are seeing Elasticsearch errors, not Kibana errors. The number you're showing me is not in the standard epoch_millis format, it's in epoch_seconds format. Take a look at the documentation here: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html
Thanks @wylie for your prompt responses but it seems that I am still missing something:
So I ran the following command in Kibana but did not resolve the issue:
PUT fscd-flat-2020-09-07
{
"mappings": {
"properties": {
"UploadTimestamp": {
"type": "date",
"format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ"
}
}
}
}
Results:
{
"fscd-flat-2020-09-07" : {
"mappings" : {
"dynamic" : "true",
"_source" : {
"includes" : ,
"excludes" :
},
"dynamic_date_formats" : ,
"date_detection" : true,
"numeric_detection" : true
}
}
}
I am still learning Kibana 
. What would it be your suggestion to resolve this issue?
Thanks in advance.
The information you've shared so far doesn't help me to understand where the issue is happening. Please share an example date that you are trying to store in Elasticsearch, and let's go from there.
date data in both JSON and Table format in Kibana 7:
"mac24": "xxxxxx",
"sw_ipport_desc": "xxxxxxxxxxx",
"sw_voip_port": "false",
**"UploadTimestamp": "1599592944165",**
"CustVertical": "xxxxxxxx",
"sw_port_desc": "fa10",
This is date format I would like to see in Kibana:
UploadTimestamp September 7th 2020, 15:52:20.554
Please let me know if you need any additional info
Using this example:
Your index can use the standard date mapping because it's in the format epoch_millis. Do not change the date format unless you need to.
You can verify that your documents are correctly understood by Elasticsearch by asking it to print the data in another date format, like this:
POST demo-timestamps-2/_search
{
"_source": true,
"fields": [{
"field": "UploadTimestamp",
"format": "date"
}]
}
To change the format in Kibana you can either change the Kibana settings which you showed above, or:
You can change it for just the field in the index pattern you care about:
Finally, it seems like you have gotten your data into a bad state by messing with this. To test this I reproduced the following scenario:
PUT demo-timestamps
PUT demo-timestamps/_mapping
{
"properties": {
"UploadTimestamp": {
"type": "date",
"format": "epoch_second"
}
}
}
PUT demo-timestamps/_doc/1
{
"UploadTimestamp": 1599592944165
}
POST demo-timestamps/_search
{
"_source": true,
"fields": [{
"field": "UploadTimestamp",
"format": "date"
}]
}
PUT demo-timestamps-2
PUT demo-timestamps-2/_mapping
{
"properties": {
"UploadTimestamp": {
"type": "date"
}
}
}
POST _reindex
{
"source": {
"index": "demo-timestamps"
},
"dest": {
"index": "demo-timestamps-2"
}
}
THANKS for your suggestion. It worked for the existing index, once new index are uploaded to Kibana, I get this new problem:
That error message is pretty clear to me: one of your indices is in a bad state, so try out the reindex method I showed above to create a copy with the correct mappings
You're are correct, when I reindex it and delete the 'bad' index the issue goes away. is there a way to make it automated? The same data goes to Kibana 6 (configured previously by another team) and there is no issue there.
I think your approach of setting the index template from your first post would work once your existing indices match
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
