Target month name to @timestamp

Elastic Stack Logstash
1

Hi everyone,

I'm using logstash to manage my catalina logs.
The issue I encounter is the fact that I have to type of timestamp in my logs.

  1. 2017-04-04 10:16:54,297
  2. 04-Apr-2017 10:16:54.443

Here my filter configuration

filter {
if [type] == "catalina"{
if [message] !~ /(.+)/ {
drop { }
}
grok {
match => [
"message", "%{TOMCAT_DATESTAMP2:timestamp} %{GREEDYDATA:ActiveThread} %{LOGLEVEL:loglevel} %{USERNAME:auth}? %{USERNAME:ident}? %{USERNAME:ident}? %{IP:clientip}? %{NOTSPACE:request}? [%{GREEDYDATA:service}] %{GREEDYDATA:message}",
"message", "%{DATESTAMP2:timestamp} %{LOGLEVEL:loglevel} [%{NOTSPACE:service}] %{GREEDYDATA:message}"
]
}
date {
match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss,SSS" ]
target => "@timestamp"
}
}
}

For my fisrt type of timestamp, I don't have any issue but for the second one I always get the tag _dateparsefailure.
Then I'm trying to find a way to convert 04-Apr-2017 to 2017-04-04 to avoid this failure.

Any ideas?

Thank you for your help.

2

Cleverly, you can put two or more potential matches in your date filter by comma separating them:

date { 
  match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss,SSS", "dd-MMM-yyyy HH:mm:ss,SSS" ]
  # target => "@timestamp" # You don't need this, it's the default behavior
  remove_field => [ "timestamp" ] 
  # This will automatically remove the now redundant `timestamp` field if the date filter 
  # successfully converts.  It will leave the field alone if it can't convert.
}

The first match wins, so I would put whichever format occurs more frequently first to reduce second checks.

3

I've tried it and I still have the date parse failure
"type" => "catalina",
"timestamp" => "04-Apr-2017 10:16:54.497",
"tags" => [
[0] "_dateparsefailure"

Does the timestamp has a specific format?

4

My bad! It's a cut and paste error on my part. The second timestamp has a decimal, not a comma, separating the milliseconds. Try this:

date { 
  match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss,SSS", "dd-MMM-yyyy HH:mm:ss.SSS" ]
  # target => "@timestamp" # You don't need this, it's the default behavior
  remove_field => [ "timestamp" ] 
  # This will automatically remove the now redundant `timestamp` field if the date filter 
  # successfully converts.  It will leave the field alone if it can't convert.
}

(I fixed the decimal).

1 Like
5

Thanks a lot Aaron you fixed my problem.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.