ES 7.4 Field [start_time] of type [keyword] does not support custom formats

Elastic Stack Elasticsearch
1

Hi, I have an elasticsearch index with dynamic field mapping. I am inserting a log into the index with a field start_time in the RFC3339 format (e.g. "2020-07-15T23:45:16Z") and until today this field was getting discovered as a date.

However in the new index we created today, it was discovered as a keyword instead. We did just upgrade to ES 7.4, but it's not clear to me why that would have affected anything.

Now when we try to run queries in Kibana, they fail and we see the error Field [start_time] of type [keyword] does not support custom formats

Is there a good way to make sure this doesn't happen? I'm also not understanding the failure. I get that it could cause queries that use the field to give unexpected results, but I don't know why it would be "unsupported", or why it would cause queries that don't involve that field to fail.

I'd really appreciate some advice, thank you!

2

Do you supply a template with mapping for this field? That is the only safe way to ensure the proper field type, probably strict_date_optional_time.

If there is no supplied mapping, Elasticsearch tries to dynamically determine the field type based on the first encountered field contents. For example a log with CSV style headers will usually map all field as keyword because it encounters the header text first.

3

I see. We do supply a template, but it does not have this field in the mapping. Unfortunately we use that template for multiple indices, some of which don't need that field.

But it sounds like I should either add the field to the template and just accept that some indices will get a useless field OR make two templates, one for each set of indices.

It's also weird that we saw a change to the field type today since the data we were sending shouldn't have changed (and it's JSON, not CSV), but I guess for now it's not worth spending too much time on getting the dynamic stuff to work when we can just explicitly specify like you suggested.

Thanks!

4

@ghirsch

Which version did you upgrade from?

You can define another template with stricter index_pattern and define just this field in that. If both patterns match mappings and settings will be merged. If you want to redefine field you can use order property. See Put index template API | Elasticsearch Guide [7.4] | Elastic

5

From 6.3. And cool, I didn't realize that was a feature! I'll look into doing that sort of multiple-template setup.

6

@ghirsch
ES switched from Joda time to Java time in 7.0. https://www.elastic.co/guide/en/elasticsearch/reference/7.x/migrate-to-java-time.html . Just FYI... Not saying it's a root cause of your issue.

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.