Logstash: Problem in date converting

Elastic Stack Logstash
1

A line from my logfile:

63001,MAt,1420396131,20-SEP-11 12.12.38.000000 AM,RecordSource,null,Can not be null,null

I want to convert --> 20-SEP-11 12.12.38.000000 AM into date time format and put it into @timestamp field.

Currently I am using this code, and name of my 4th column is "Date"

date {
match => ["Date", "dd-MMM-YY HH.mm.ss.SSSSS"]
target => "@timestamp"

}

and I get the following message:

{
"message" => "299001,MAt,1322737460,20-SEP-11 02.25.45.000000 PM,ClassificationType,null,Can not be null,null",
"@version" => "1",
"@timestamp" => "2016-05-26T23:41:55.767Z",
"path" => "/Users/paritp/Desktop/SampleErrors.txt",
"host" => "685b358426d6.ant.amazon.com",
"type" => "core2",
"ID KEY" => 299001,
"Client ID" => "MAt",
"Account ID" => 1322737460,
"Date" => "20-SEP-11 02.25.45.000000 PM",
"Error Field Name" => "ClassificationType",
"Error Field Value" => "null",
"Error Message" => "Can not be null",
"Record Source" => "null",
"tags" => [
[0] "_dateparsefailure"
]
}

Please help.

2

"tags" => [
[0] "_dateparsefailure"
]

Why am I getting this?

3

This is my config file

input {
file {
path => "/Users/paritp/Desktop/SampleErrors.txt"
type => "core2"
start_position => "beginning"
ignore_older => 0
}
}
filter {

csv {
separator => ","
columns => ["ID KEY","Client ID","Account ID","Date","Error Field Name","Error Field Value","Error Message", "Record Source"]
autogenerate_column_names => false

  convert => {"ID KEY" => "integer"}
  convert => {"Account ID" => "integer"}
  convert => {"Date" => "date_time"}

}

date {
match => ["Date", "dd-MMM-YY HH.mm.ss.SSSSSS"]
remove_field => "Date"

}

}

output {
elasticsearch {
action => "index"
hosts => "localhost"
index => "samperror"
workers => 1
}
stdout {
codec => rubydebug
}

}

4

"Date" => "20-SEP-11 02.25.45.000000 PM",

match => ["Date", "dd-MMM-YY HH.mm.ss.SSSSSS"]

You're ignoring that it's a 12-hour time. You should try "hh.mm.ss.SSSSSS a" or maybe "KK.mm.ss.SSSSSS a".

The uppercase month name might also be a problem. IIRC the date filter (actually the Joda-Time library) is annoyingly case-sensitive.

1 Like
5

Thanks Magnus. It worked. :slight_smile: