Date filter plugin doesn't take the timezone into account when using ISO8601

Hi, I'm trying to normalize some timestamps into UTC from CEST, so far I've had no issues whatsoever except for when the parsed timestamp is already in ISO8601. At first, I wasn't sure whether the date-filter-plugin was working correctly so I tried the following:

mutate {
		add_field => { "timestamp" => "%{@timestamp}" }
		remove_field => ["@timestamp"]
	}

And for the date plugin I have this:

date {
        # timezone => "UTC"
        timezone => "Etc/GMT-2"
        target => "@timestamp"
        match => [
            "timestamp",
            "ISO8601"
        ]
        remove_field => [ "timestamp" ]
}

I'm parsing json logs directly and therefore I also parse the "@timestamp" as following:

{ "@timestamp": "2020-11-25T18:34:09.542Z"}

The resulting timestamp should be 2h behind the one parsed, but so far I'm getting the same timestamp (i.e: 2020-11-25T18:34:09.542Z).

Any idea if this is a bug or am I doing something wrong?

If the timezone is specified in the string you are trying to parse then you cannot override it using the timezone option.

The Z at the end of the string indicates it is in Zulu time, also known as UTC. If you think that Z should not be there then you can remove it using

mutate { gsub => [ "timestamp", "Z$", "" ] }
1 Like

How does your original timestamp looks like? Can you share an example?

As badger said, if your original timestamp has an Z at the end, it is already in UTC and the timezone option from the date filter will do nothing.

If your timezone is in local time but wrongly has a Z at the end, then you can use the gsub provided to remove it.

1 Like

Thank you very much to both of you. It was the Z thingy, I didn't realise it could cause this "issue" but it makes a lot of sense. I ended up using the gsub line and is all good now