Kibana can't work with Elasticsearch explicit date format as input

Elastic Stack Kibana
1

Hello.

In the company that I work for, we are currently using Elasticsearch 8.6.2 + Kibana 8.6.2 (both with authentication enabled) for create a custom dashboard where we are showing some metrics that we need.

As the Timestamp field of our data view, we are using a custom field called initialTime.

From kibana, we are setting a custom format to the mentioned field, YYYY-MM-DD'T'HH:mm:ss.SSSSSSS

From our .NET6 API we are sending data to our Elasticsearch index. When the input of the initialTime date came with this format (YYYY-MM-DD'T'HH:mm:ss.SSSSSSS), all works as expected and we can use in our dashboard the time filter. The problem is that we are also sending some data where the input of the initialTime date came also with the timezone, that is, this format: YYYY-MM-DD'T'HH:mm:ss.SSSSSSSZ. When this happen, we can't see the data that came with this format (YYYY-MM-DD'T'HH:mm:ss.SSSSSSSZ), and we have to deactivate the Timestamp field in our data view in order to be able to see all the data from the dashboard.

If we create the initialTime field in Elasticsearch with a custom format (YYYY-MM-DD'T'HH:mm:ss.SSSSSSS), we can't see any data as happen with the last described case (we think that this happen because the mentioned format is not valid as the input that kibana expects to receive).

So now what we are doing is mapping our data from code (C# with .NET6) before send it to Elasticsearch index and removing the timezone. By doing this and by not setting any format to initialTime field in Elasticsearch index, we are getting the desired behaviour and we are able to filter our data from Kibana dashboard.

We are trying to avoid this explicit conversion from code but we didn't find any useful information after check elastic topics, stackoverflow...

Does someone have any thoughts about this?

Please feel free to request any information that you could need.

4

Not sure if I understood all the problem but I believe you can define some formats for the date field using Date field type | Elasticsearch Guide [8.8] | Elastic

Would that work for you?

5

Hello @dadoonet, thanks for your response.

Even if we set multiple formats for the date field, we can't indicate multiple formats from Kibana UI. So we won't be able to format the received input in kibana and we won't be able to use the mentioned initialTime as the timestamp field of our data view and without that, we can't use the date filter from our Kibana dashboard

6

So there is no issue on Elasticsearch side, right? You are able to deal with both formats already?

7

Yes. Even if we dont create the elasticsearch index with custom map (from c# model), when we insert the first element, the initialTime is created as date without format and we can send (and insert into the elasticsearch index) both format (with and without timezone (gmt) dates)

8

Ok. So I'm moving the discussion under #elastic-stack:kibana.

I don't understand what's the problem on the Kibana side. May be some screen captures would help. Unless the community who is more involved in Kibana knows it :wink:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.