I have an ELK stack running and at some point it starts throwing: "IllegalArgumentException[Invalid format: "Mar 20 02:00:06"];" After restarting logstash it works fine again. When I try to parse the line that gave the error again, it parses it just fine. Is it an ES or logstash bug?
Example log line to parse:
"Mar 21 01:59:59 136.244.34.211 INFO krdxStats ,0,,sobasnvved1ure6l7683a6djo2,com_krdx_search,,,report,10753953,67.249.78.11,67.249.78.11,2016-03-21 1:59:59,krdxCompanyDetails,getDetails,cnt:1"
Grok pattern:
filter {
if [type] == "project" {
grok {
match => [ "message", "(?%{MONTH} (?:(?:[ 0][1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) %{TIME}) %{IP:app_ip} %{LOGLEVEL:level} %{WORD:logger} ,%{NUMBER:username},(?:%{DATA:user_fullname}|),(?:%{DATA:cookie}|),(?:%{DATA:web_app_component}|),(?:%{DATA:}|),(?:%{DATA:controller}|),(?:%{DATA:view}|),(?:%{DATA:query_subject_id}|),(?:%{IP:client_ip}|),(?:%{IP:}|),(?:%{TIMESTAMP_ISO8601:method_call_time}|),(?:%{WORD:class_name}|),(?:%{WORD:method_name}|)(?:,%{GREEDYDATA:method_response}|)" ]
}
date {
match => [ "method_call_time" , "yyyy-MM-dd' 'HH:mm:ss" ]
timezone => "Europe/Tallinn"
}
}
}
ES log:
[2016-03-19 23:59:59,999][DEBUG][action.bulk ] [Venus Dee Milo] [logstash-project-2016.03.20][1] failed to execute bulk item (index) index {[logstash-project-2016.03.20][project][AVORUj5u9MXhDshT2kQv], source[{"message":"Mar 20 02:00:06 136.244.34.211 INFO krdxStats ,0,,vve7g2tohjjnja055c6tb112g6,com_krdx_search,,,comments,11940893,15.255.253.93,15.255.253.93,2016-03-20 2:00:06,krdxPersonMain,getPersonData,dataType:avatar,cnt:1","@version":"1","@timestamp":"2016-03-20T00:00:06.000Z","beat":{"hostname":"project","name":"project"},"count":1,"fields":null,"input_type":"log","offset":322435950,"source":"/opt/log/project/2016/03/20/project.log","type":"project","host":"project","timestamp":"Mar 20 02:00:06","app_ip":"136.244.34.211","level":"INFO","logger":"krdxStats","username":"0","cookie":"vve7g2tohjjnja055c6tb112g6","web_app_component":"com_krdx_search","view":"comments","query_subject_id":"11940893","client_ip":"15.255.253.93","method_call_time":"2016-03-20 2:00:06","class_name":"krdxPersonMain","method_name":"getPersonData","method_response":"dataType:avatar,cnt:1"}]}
MapperParsingException[failed to parse [timestamp]]; nested: IllegalArgumentException[Invalid format: "Mar 20 02:00:06"];
Logstash log:
{: timestamp => "2016-03-20T23:59:53.306000+0000", : message => "Failed action. ", : status => 400, : action => ["index", {: _id => nil,
: _index => "logstash-project-2016.03.20",
: _type => "project",
: _routing => nil
..... lot more stuff here ....
>> ], : response => {
"create" => {
"_index" => "logstash-project-2016.03.20", "_type" => "project", "_id" => "AVOWeHM-9MXhDshTKxOr", "status" => 400, "error" => {
"type" => "mapper_parsing_exception", "reason" => "failed to parse [timestamp]", "caused_by" => {
"type" => "illegal_argument_exception", "reason" => "Invalid format: "Mar 21 01:59:59""
}
}
}
}, : level => : warn
}