Date format in Elasticsearch

Elastic Stack Logstash
1

Hello. I am trying to import a .csv file to Elasticsearch. In the .csv file there is a date field in the "yyyyMMdd" format (for example 20220326). This is what I have in my logstash conf file:

filter {
       date {
		match => ["date", "yyyyMMdd"]
		target => "date"
	}
}

After I imported the file I realized that it saves the date one day early, for example if I had the date 20220326 in the .csv file, this is what I get in Elasticsearch (on Kibana console): 2022-03-25T23:00:00.000Z.

How can I solve this? Thank you in advance.

2

Elasticsearch will always store date with UTC timezone
Provide timezone in your logstash filter as follow

filter {
       date {
		match => ["date", "yyyyMMdd"]
		target => "date"
        timezone => "+01:00"
	}
}
3

Thank you for the reply. This didn't really solve my problem - it still saves he wrong day. Is there a way to save the date without time? For example, to be just 2022-03-26?

4

If you want the field on the document to be a date, then it is stored as milliseconds since the epoch, so it cannot store the date without the time.

If you want a string in a different format then this shows an example.

If you just want the date why are you using a date filter at all?

5

The thing is, i need to import the file to an index that is already created with date fields. But if i import the file without the date filters, when i call the query i get the date as 20210362 which is good but in Kibana Discover it saves the date as "+20210326-01-01T00:00:00.000Z" and i'm not able to create any visualizations. Maybe the problem is in Kibana and i can manually change the format there?

6

You can change the timezone in Kibana Advanced Settings. Default is it uses what the browser detects. Swap to UTC and see if that works for you.

image
image2108×974 202 KB

7

Sorry btw i'm very new at Elasticsearch. In my case that doesn't really change anything, cause it saves the whole date as the year aka 20210326-01-01T00:00:00.000Z instead of 2021-03-26T00:00:00.000Z (without using the date filters that i mentioned above)

8 
PUT timezone
{
  "mappings": {
    "properties": {
      "date": {
        "type":   "date",
        "format": "yyyyMMdd"
      }
    }
  }
}

POST timezone/_doc
{
  "date": "20220326"
}

GET timezone/_search
#notice the date field shows exactly what you entered. No conversion.

Kibana in Discovery and Visualizations datetime conversion happens for each user based on the settings in Kibana.

Discover date field = Mar 25, 2022 @ 20:00:00.000 with timezone setting to equal browser. I am -4 hours from UTC.

Discover date field = Mar 26, 2022 @ 00:00:00.000 with timezone setting to UTC.

I think you might be missing the part where Kibana is auto translating timestamps based on your Kibana settings.

9

I get it now. Thank you very much!

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.