Logstash timestamp shift

Elastic Stack Logstash
1

Hi !

I came across a strange behavior while parsing a timestamp epoch style


    date {
    match => [ "eventtime_ms","UNIX" ]
    target => "[event][created]"
    timezone => "Etc/GMT+2"
    }
        date {
    match => [ "eventtime_ms","UNIX" ]
    target => "[event][created4]"
    timezone => "Etc/GMT+4"
    }

Using this 2 date filters i came across the same time while viewing in kibana ?

I do understand that Kibana can shit timezones automatically when viewing date types but the JSON view tells me that nothing is happening


   "event": {
      "created4": "2023-04-20T14:15:27.000Z",
      "created": "2023-04-20T14:15:27.000Z"

I think i'm missing something here ! What's the catch ?

2

time is same. but it has TZ attached now.

3

Where can i see that ?

4

put that in elasticsearch,
create two pattern (data view) one with created4 and created.
now draw graph and you will see difference

5

I would expect the date filter to ignore the timezone option when parsing UNIX or UNIX_MS. Those are intervals since the epoch in GMT, so they convert to the same UTC time in every timezone.

6

This imply that epoch is GMT by default, this makes sense !

This means that i have different timezones from thoses logs i have to shift myself from the [event][created] source with ISOXXXX format ?

7

If the source logs have a timezone offset on an epoch based timestamp then they are not actually UNIX or UNIX_MS. You could parse them as that, then convert to a string, mutate+gsub to remove the "Z$", and then go through the date filter again with the timezone set.

1 Like
8

I got you ! It's actually fortinet logs with nanoseconds precision.

Actually i'd like to know if grok is the best solution to substract char from a string since the timestamp i'm working with is nanosecond precision.

    grok {
        match => {
            eventtime => "(?<eventtime_ms>..........)"

        }

Since you suggested mutate + gsub to remove Zulu marking there is maybe a better solution...

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.