Date filter plugin doesn't take the timezone into account when using ISO8601

[Jose_E (José Esteban Infantes)]

Hi, I'm trying to normalize some timestamps into UTC from CEST, so far I've had no issues whatsoever except for when the parsed timestamp is already in ISO8601. At first, I wasn't sure whether the date-filter-plugin was working correctly so I tried the following:

mutate {
		add_field => { "timestamp" => "%{@timestamp}" }
		remove_field => ["@timestamp"]
	}

And for the date plugin I have this:

date {
        # timezone => "UTC"
        timezone => "Etc/GMT-2"
        target => "@timestamp"
        match => [
            "timestamp",
            "ISO8601"
        ]
        remove_field => [ "timestamp" ]
}

I'm parsing json logs directly and therefore I also parse the "@timestamp" as following:

{ "@timestamp": "2020-11-25T18:34:09.542Z"}

The resulting timestamp should be 2h behind the one parsed, but so far I'm getting the same timestamp (i.e: 2020-11-25T18:34:09.542Z).

Any idea if this is a bug or am I doing something wrong?

[Badger]

If the timezone is specified in the string you are trying to parse then you cannot override it using the timezone option.

The Z at the end of the string indicates it is in Zulu time, also known as UTC. If you think that Z should not be there then you can remove it using

mutate { gsub => [ "timestamp", "Z$", "" ] }

[leandrojmp (Leandro Pereira)]

How does your original timestamp looks like? Can you share an example?

As badger said, if your original timestamp has an Z at the end, it is already in UTC and the timezone option from the date filter will do nothing.

If your timezone is in local time but wrongly has a Z at the end, then you can use the gsub provided to remove it.

[Jose_E (José Esteban Infantes)]

Thank you very much to both of you. It was the Z thingy, I didn't realise it could cause this "issue" but it makes a lot of sense. I ended up using the gsub line and is all good now