Replace timestamp from logfile - dateparsefailure

Elastic Stack Logstash
1

Hi,

my logs starting with:

10.01.2018 | 07:43:13,961 | INFO |
09.01.2018 | 23:59:02,119 | DEBUG |

I tried following:

%{DATE_EU:Datum} | (%{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}%{ISO8601_TIMEZONE:timezone})

But hour, minute etc are empty (null)

How could I replace timestamp with this Date?

Regards

2

Something like (?<date>[\d.\d.\d]{10}) (?<time>[\d:\d:\d,\d]{12}) should work but in the grok debugger the | doesn't play nice so you might want to use a gsub filter first to remove/replace that character.

Then use mutate to create a new field
mutate { add_field => { "TIMEDATE" => "%{date} %{time}" } }

And date to replace it with the timestamp
date {
match => [ "TIMEDATE", "yyyy-MM-dd HH:mm:sss" ] (change this)
timezone => "UTC"
target => "@timestamp"
}

Not pretty but it will work.

3

Thank you,

in grogdebugger I got matches for:

09.01.2018 | 23:59:02,119 | DEBUG |

Pattern:

(?[\d.\d.\d]{10}) | (?[\d:\d:\d,\d]{12})

Now I created following config file (for Test I created an input file)

input {
file {
path => ["/tmp/app"]
type => "FilePluginInput"
start_position => beginning
sincedb_path => "/dev/null"
}
}

filter {
if [type] in [ "FilePluginInput" ] {
grok {
mutate {
gsub => [
"message", "|", "", // => remove |
]
}
match => [
"message" , "(?[\d.\d.\d]{10}) | (?[\d:\d:\d,\d]{12})"
]
}
mutate {
add_field => { "TIMEDATE" => "%{date} %{time}" }
add_tag => [app]
}
date {
match => [ "TIMEDATE", "yyyy-MM-dd HH:mm:sss" ]
timezone => "UTC"
target => "@timestamp"
}
}
}

Bevor | there is a backslash , this one isnt displayd in my thread entry.
Any Idea?

4

I think I'm nearly to finish. Now I got dateparse faillure:

Following Fields:

my filter:

input {
file {
path => ["/tmp/app"]
type => "FilePluginInput"
start_position => beginning
sincedb_path => "/dev/null"
}
}

filter {
grok {
match => [ "message" , "%{DATE_EU:Datum} | (?[\d:\d:\d,\d]{12})" ]
}
mutate {
add_field => { "TIMEDATE" => "%{Datum} %{Uhrzeit}" }
}
date {
match => [ "TIMEDATE", "YYYY-MM-dd'T' HH:mm:ss.SSSZ" ]
target => "@timestamp"
}
}

Fields:
"TIMEDATE" => "10.01.2018 13:37:02,942",
"path" => "/tmp/app",
"@timestamp" => 2018-01-10T12:45:42.417Z,
"Datum" => "10.01.2018",
"@version" => "1",
"host" => "XXX",
"message" => "10.01.2018 | 13:37:02,942 | DEBUG | yyyyy",
"type" => "FilePluginInput",
"Uhrzeit" => "13:37:02,942",
"tags" => [
[0] "_dateparsefailure"

Regards

5

It's an hardway :wink: Now I think I'm at 99% finished.

My Filter:

filter {
grok {
match => [ "message" , "%{DATE_EU:Datum} | (?[\d:\d:\d,\d]{12})" ]
}
mutate {
add_field => { "TIMEDATE" => "%{Datum} %{Uhrzeit}" }
}
date {
timezone => "Europe/Berlin"
match => [ "TIMEDATE", "ISO8601", "dd.MM.yyyy HH:mm:ss','SSSS" ]
target => "@timestamp"
}
}

Output:

  "TIMEDATE" => "09.01.2018 23:59:02,119",
      "path" => "/tmp/app",
"@timestamp" => 2018-01-09T22:59:02.119Z,
     "Datum" => "09.01.2018",
  "@version" => "1",

Timestamp is 1 hour earlier than TIMEDATE, what can I do?

6

Elastic always stores times as UTC. You told it that timestamp was in the Europe/Berlin timezone, which is UTC +1:00, so this is working as designed.

7

thank you,

in the meantime I also found it.

thanks everyone for help.

Regards

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.