Failed parsing date from field using logstash

Elastic Stack Logstash
1

Hi All,

I trying to index sample log files to ES using logstash and below is the logstash conf file,

input {
file {
type => "apache"
path => "/opt/esadmin/elasticsearch-1.7.3/bin/logstash-1.5.5/logging_AdminServer.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter {
grok {
match => [ "message", "%{MONTHDAY:monthday}-%{MONTH:month}-%{YEAR:year}\s*%{TIME:StartTime}\s*%{TIME:EndTime}\s*%{TIME:ElapseTime}\s*%{WORD:External_Itemcode}\s*%{NUMBER:User_id} %{GREEDYDATA:Event_name}" ]
}

mutate {
   add_field => { 
     "Date" => "%{monthday}-%{month}-%{year}"
   }
}
date {
   match => [ "Date" , "dd-MMM-yyyy" ]
   locale => "en"
}
mutate {
   remove_field => [ "month", "monthday", "year"]
}

}

output {
elasticsearch {
host => "10.8.157.24:9200"
protocol => "http"
user => "esadmin"
password => "password"
}
stdout {
codec => "dots"
}
}
and I am getting the following error response when I executing logstash.conf file,

Failed parsing date from field {:field=>"Date", :value=>"%{monthday}-%{month}-%{year}", :exception=>"Invalid format: "%{monthday}-%{month}-%{year}"", :config_parsers=>"dd-MMM-yyyy", :config_locale=>"en", :level=>:warn}
Logstash startup completed
.....................
Logstash shutdown completed

Please let us know your suggestions.

Thanks,
Ganeshbabu R

2

What does your input data look like?

3

Hi @warkolm,

Below is the sample input data,

Date StartTime EndTime ElapseTime External_Itemcode User_id Event_name
26-May-2016 05:11:25 05:12:50 00:01:25 0124524554 3024 Loading loadItemstoWorkspace Details
26-May-2016 05:14:34 05:15:28 00:00:54 0124524554 3024 Loading loadItemstoWorkspace Details
26-May-2016 05:17:22 05:17:30 00:00:08 0124524554 3024 Create Nielsen Item
26-May-2016 05:18:19 05:18:46 00:00:26 0124524554 4137 Loading UpdateCharacteristicValues Details

Thanks,
Ganeshbabu R

4

Your Date field doesn't contain the date but the string "%{monthday}-%{month}-%{year}", which indicates that your grok filter failed.

5

Hi @magnusbaeck,

In our sample log data we have the header "Date" so we trying to add the date as "new field" with the format "dd-MMM-yyyy"

Is there any other workaround activity?

Thanks,
Ganeshbabu R

6

As I said, I don't think your grok filter is working properly. If you fix that the date parsing problem will be resolved too.

I suggest you temporarily replace your elasticsearch output with a stdout { codec => rubydebug } output until you've debugged this problem. If you show the Logstash output with this stdout output it'll be easier to help.

7

Hi @magnusbaeck

Please find the below response of the logstash output,

Failed parsing date from field {:field=>"Date", :value=>"%{monthday}-%{month}-%{year}", :exception=>"Invalid format: "%{monthday}-%{month}-%{year}"", :config_parsers=>"dd-MMM-yyyy", :config_locale=>"en", :level=>:warn}
Logstash startup completed
{
"message" => "Date StartTime EndTime ElapseTime External_Itemcode User_id Event_name",
"@version" => "1",
"@timestamp" => "2016-05-31T09:04:25.956Z",
"host" => "dayrhebfmd001.enterprisenet.org",
"path" => "/opt/esadmin/elasticsearch-1.7.3/bin/logstash-1.5.5/logging_AdminServer.log",
"type" => "apache",
"tags" => [
[0] "_grokparsefailure",
[1] "_dateparsefailure"
],
"Date" => "%{monthday}-%{month}-%{year}"
}
{
"message" => "26-May-2016\t05:11:25 05:12:50 00:01:25 0124524554 3024 Loading loadItemstoWorkspace Details",
"@version" => "1",
"@timestamp" => "2016-05-31T09:04:25.958Z",
"host" => "dayrhebfmd001.enterprisenet.org",
"path" => "/opt/esadmin/elasticsearch-1.7.3/bin/logstash-1.5.5/logging_AdminServer.log",
"type" => "apache",
"StartTime" => "05:11:25",
"EndTime" => "05:12:50",
"ElapseTime" => "00:01:25",
"External_Itemcode" => "0124524554",
"User_id" => "3024",
"Event_name" => " Loading loadItemstoWorkspace Details",
"Date" => "2016-05-26T04:00:00.000Z"
}
{
"message" => "26-May-2016\t05:14:34 05:15:28 00:00:54 0124524554 3024 Loading loadItemstoWorkspace Details",
"@version" => "1",
"@timestamp" => "2016-05-31T09:04:25.958Z",
"host" => "dayrhebfmd001.enterprisenet.org",
"path" => "/opt/esadmin/elasticsearch-1.7.3/bin/logstash-1.5.5/logging_AdminServer.log",
"type" => "apache",
"StartTime" => "05:14:34",
"EndTime" => "05:15:28",
"ElapseTime" => "00:00:54",
"External_Itemcode" => "0124524554",
"User_id" => "3024",
"Event_name" => " Loading loadItemstoWorkspace Details",
"Date" => "2016-05-26T04:00:00.000Z"
}
{
"message" => "",
"@version" => "1",
"@timestamp" => "2016-05-31T09:04:25.960Z",
"host" => "dayrhebfmd001.enterprisenet.org",
"path" => "/opt/esadmin/elasticsearch-1.7.3/bin/logstash-1.5.5/logging_AdminServer.log",
"type" => "apache",
"tags" => [
[0] "_grokparsefailure",
[1] "_dateparsefailure"
],
"Date" => "%{monthday}-%{month}-%{year}"
}

failed action with response of 400, dropping action: ["index", {:_id=>nil, :_index=>"logstash-2016.05.31", :_type=>"apache", :_routing=>nil}, #<LogStash::Event:0x5e14189f @metadata_accessors=#<LogStash::Util::Accessors:0x7d9d82c1 @store={"path"=>"/opt/esadmin/elasticsearch-1.7.3/bin/logstash-1.5.5/logging_AdminServer.log", "retry_count"=>0}, @lut={"[path]"=>[{"path"=>"/opt/esadmin/elasticsearch-1.7.3/bin/logstash-1.5.5/logging_AdminServer.log", "retry_count"=>0}, "path"]}>, @cancelled=false, @data={"message"=>"Date StartTime EndTime ElapseTime External_Itemcode User_id Event_name", "@version"=>"1", "@timestamp"=>"2016-05-31T09:04:25.956Z", "host"=>"dayrhebfmd001.enterprisenet.org",
"path"=>"/opt/esadmin/elasticsearch-1.7.3/bin/logstash-1.5.5/logging_AdminServer.log", "type"=>"apache", "tags"=>["_grokparsefailure", "_dateparsefailure"], "Date"=>"%{monthday}-%{month}-%{year}"}, @metadata={"path"=>"/opt/esadmin/elasticsearch-1.7.3/bin/logstash-1.5.5/logging_AdminServer.log", "retry_count"=>0}, @accessors=#<LogStash::Util::Accessors:0x4c2fef07 @store={"message"=>"Date StartTime EndTime ElapseTime External_Itemcode User_id Event_name", "@version"=>"1", "@timestamp"=>"2016-05-31T09:04:25.956Z", "host"=>"dayrhebfmd001.enterprisenet.org", "path"=>"/opt/esadmin/elasticsearch-1.7.3/bin/logstash-1.5.5/logging_AdminServer.log", "type"=>"apache", "tags"=>["_grokparsefailure", "_dateparsefailure"], "Date"=>"%{monthday}-%{month}-%{year}"}, @lut={"host"=>[{"message"=>"Date StartTime EndTime ElapseTime External_Itemcode User_id Event_name", "@version"=>"1", "@timestamp"=>"2016-05-31T09:04:25.956Z", "host"=>"dayrhebfmd001.enterprisenet.org",

Pls let us know your feedback..

Thanks,
Ganeshbabu R

8

The grok filter and your date filter work just fine for correct input (the second and third messages) but fails for empty lines and the first line which contains a list of column headers. This is expected. Perhaps you should drop the column headers and empty lines using a drop filter that you wrap in a conditional.