Logstash date filter

Elastic Stack Logstash
1

Hi i have a short_date field in the following format

09/Aug/2023:12:44:15 +0530

This field is created as text. To convert it to date i am doing the following

      date {
         match => [ "short_date", "dd/MMM/yyyy:HH:mm:ss Z" ]
         target=> "short_date"
      }

The field is getting saved as date but the time value is coming different
for the date 09/Aug/2023:12:44:15 +0530
the value is coming as 2023-08-09T07:14:15.000Z
Timing is different. Where is the mistake>

2

Hello Neelam,

There is no error: I guess your local time is India Standard Time as the date shows +0530 and Elastic stores the dates always in the Zulu Time (Z). So your date was parsed correctly and then translated into the standard timezone.

In Kibana, you can configure the timezone that should be shown under advanced settings dateFormat:tz so you can view the date as you expect.

Best regards
Wolfram

3

Thanks for the quick response. but the default @timestamp field in every document comes in correct time

4

The default @timestamp relies on the systems date settings I think so it can be system specific.

Are you running on Windows or Linux? Is the systems timezone configured correctly?

5

it's on linux.

image
image1567×671 58.9 KB

This is the kibana setting
What do i need to change to have same date value in short_date? can i specify the timezone in the date filter?

6

Yes, you can configure the date filter with a timezone but I don't know if this is used when the date already has a timezone given.

7

one observation. I have the same setup on a windows machine too.
But on that machine the date format is slightly different it has mili seconds too

09/Aug/2023:13:39:31.066 +0530

so after i apply the date filter

      date {
         match => [ "short_date", "dd/MMM/yyyy:HH:mm:ss Z" ]
         target=> "short_date"
      }

since the :SSS is missing short_Date is still in text format but the value of time is correct

09/Aug/2023:13:39:31.066 +0530

Is it not getting converted at all?

8

I guess this is true - when a date expression did not match this does not cause an error.

You should be able to check this easily: By default, the date filter adds a tag when the parsing fails: Date filter plugin | Logstash Reference [8.9] | Elastic

So you can check if this document has a _dateparsefailure tag (if you didn't remove the tags)

9

yes it had, when i added a .SSS to the date filter match pattern it again works the same way as it was in linux m/c. the time changes. I'm still not clear as to what has to be changed?

10

If anything, you should change Kibana timezone to show the date correctly. The ingest works correctly as it aligns incoming dates to a common timezone (Zulu) so that you can compare dates in docs from different countries(e.g. India and US).

11

Hi, got it working. i added timezone=> "Universal" in the date filter. and now the time is matching. Thank you for the quick support!!

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.