Hello.
In the company that I work for, we are currently using Elasticsearch 8.6.2 + Kibana 8.6.2 (both with authentication enabled) for create a custom dashboard where we are showing some metrics that we need.
As the Timestamp field of our data view, we are using a custom field called initialTime.
From kibana, we are setting a custom format to the mentioned field, YYYY-MM-DD'T'HH:mm:ss.SSSSSSS
From our .NET6 API we are sending data to our Elasticsearch index. When the input of the initialTime date came with this format (YYYY-MM-DD'T'HH:mm:ss.SSSSSSS), all works as expected and we can use in our dashboard the time filter. The problem is that we are also sending some data where the input of the initialTime date came also with the timezone, that is, this format: YYYY-MM-DD'T'HH:mm:ss.SSSSSSSZ. When this happen, we can't see the data that came with this format (YYYY-MM-DD'T'HH:mm:ss.SSSSSSSZ), and we have to deactivate the Timestamp field in our data view in order to be able to see all the data from the dashboard.
If we create the initialTime field in Elasticsearch with a custom format (YYYY-MM-DD'T'HH:mm:ss.SSSSSSS), we can't see any data as happen with the last described case (we think that this happen because the mentioned format is not valid as the input that kibana expects to receive).
So now what we are doing is mapping our data from code (C# with .NET6) before send it to Elasticsearch index and removing the timezone. By doing this and by not setting any format to initialTime field in Elasticsearch index, we are getting the desired behaviour and we are able to filter our data from Kibana dashboard.
We are trying to avoid this explicit conversion from code but we didn't find any useful information after check elastic topics, stackoverflow...
Does someone have any thoughts about this?
Please feel free to request any information that you could need.
