Not sure what you are seeing, but when I say converts the time to UTC it means it will interpret the time as UTC, if the date string does not have any offset, no offset will be added.
Can you share the following:
- A sample of your log before ingesting to Elasticsearch
- A sample of the same log in Elasticsearch showing how the date field is stored (the json view in Kibana)
But for example, imagine the your original logs has the following date string on a field:
timestamp: "2023-10-13 10:00:00"
This date string does not have any information about the timezone in which it was generated, it is impossible to know if this date string has a timezone offset or not.
When Elasticsearch receives this data and if the mapping is correct, like the field was mapped as a
date field or elasticsearch was able to infer that the field is a date field, this date string will be interpreted as an UTC date, so you will end up with something like this:
Now, if you go to Kibana and your Kibana timezone is using the Sidney timezone, then this UTC date will be converted to it and you will see
The problem will be if your original date string was generated with an offset without having the offset in the date string.
Is this time
2023-10-13 10:00:00 is the same as ``2023-10-13 10:00:00+1100`, but do not have the offset in the string, then you will have issues because elasticsearch will not know about the offset and consider it an UTC time and kibana will add another offset as well.
To solve this you have two options:
- Change your logs to have the offset information in the date string
- Use a date filter during the ingestion of the data to inform about the offset
Never used NiFi, but you can use an Ingest Pipeline in Elasticsearch with the date filter to parse your date correctly, how you will do that in NiFi I'm not sure.
You can however add a setting to your index template to tell elasticsearch to always run a specific ingest pipeline.
Adding this to your template index settings will tell Elasticsearch to always run this pipeline: