Intermittent grok date parse failures

Elastic Stack Logstash
1

Hi all,

I am importing IIS logs into ES using Filebeat and Logstash, using grok to manipulate the input.

I am getting strange, intermittent date parse failures with grok (only 65 records out of over nearly 2 million) but I cannot see for the life of me what is wrong with the format as two records next to each other in the same log can parse differently and the date format is consistent.

For example:

2018-03-25 01:47:02 W3SVC1...

fails but:

2018-04-19 09:57:12 W3SVC1...

works!

There are no other date fields in the records so it can only be these fields.

My grok filter is:

filter {
if [message] =~ "^#" {
drop {}
}

if [message] =~ "^.192.168.1.1.$" {
drop{}
}

grok {
match => { "message" => "%{GREEDYDATA:log_timestamp} %{WORD:S-SiteName} %{NOTSPACE:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{NOTSPACE:CS-URI-Stem} %{NOTSPACE:cs-uri-query} %{NUMBER:S-Port} %{NOTSPACE:CS-Username} %{IPORHOST:C-IP} %{NOTSPACE:CS-Version} %{NOTSPACE:CS-UserAgent} %{NOTSPACE:CS-Cookie} %{NOTSPACE:CS-Referer} %{NOTSPACE:CS-Host} %{NUMBER:SC-Status} %{NUMBER:SC-SubStatus} %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}"}
}

date {
match => [ "log_timestamp","yyyy-MM-dd HH:mm:ss"]
timezone => "Europe/London"
target => "@timestamp"
}

which, like I say, works for pretty much every record apart from these 65!

Any ideas?

2

The time 2018-03-25 01:47:02 GMT doesn't exist since the UK entered daylight savings time on 2018-03-25 01:00. Perhaps the timestamp actually has another timezone?

3

Good spot! Looking at it, it looks like UTC. I'll give it a try! Thx

4

Looks like that did it! 700,000 records so far and not a single grok failure. Thanks for your help Magnus.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.