Need Help parsing timestamps

Elastic Stack Logstash
1

I am curently parsing azure messages. Sadly azure sends some dates in the following format:

2018-06-03 01:00:12Z

at first I thought this would be a normal ISO8601 but this did not match. I tried a Formats like yyyy-MM-dd HH:mm:ssZ and yyyy-MM-dd HH:mm:ss but this did not work also. Do I have to remove the Z at the end of the Azure timestamp?

Thanks in advance.

2

For non-formatting syntax, you’ll need to put single-quote characters around the value.

date { match => { "someField", "yyyy-MM-dd HH:mm:ss'Z'" } }
3

Even with the single quoted Z the error persists.

date {
match => [ "[records][properties][eventProperties][Start Time]", "yyyy-MM-dd HH:mm:ss'Z'" ]
}

[records.properties.eventProperties.Start Time]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: "2018-06-03 01:00:12Z" is malformed at " 01:00:12Z""}}}}}

4

Try Grok filter

%{TIMESTAMP_ISO8601:DATE}

5

Try this

date { match => { "someField", "yyyy-MM-dd HH:mm:ssZ" } }

6

That's not an exception from the date filter, it's a mapper parsing exception from elasticsearch, right?

7

Yes this was an error by elasticsearch. I dont know why, but i tested the config and viewed the output via output on stdout and everything seemed fine with the 'Z'. For now this error hasent apeared since. I will report back to you guys if it happens again.

8

If you index a field that contains "2018-06-03" then elasticsearch expects that field to contain a date. "2018-06-03 01:00:12Z" is not a date, it is a timestamp. So it will complain about the field format all day, until it rolls to a new index, at which point it starts expecting the field to contain a timestamp. (Assuming you do not have a template forcing the field format.)

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.