I am attempting to parse some syslog messages and I'm seeing the following message in the Logstash logs whenever I try to match some parsed syslog field:
:response=>{"create"=>{"_index"=>"logstash-2016.06.15", "_type"=>"logstash", "_id"=>"AVVWH4KY1Ye0QKQmJ6e1", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"2016-06-15 22:12:33,254\" is malformed at \"-06-15 22:12:33,254\""}}}}, :level=>:warn}
Here are the configs causing the problem:
if [type] == "logstash" {
grok {
match => [ "message", "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ]
}
date {
match => [ "syslog_timestamp", "ISO8601" ]
remove_field => ["syslog_timestamp"]
}
mutate {
lowercase => [ "syslog_hostname" ]
add_field => { "environment" => "%{syslog_hostname}" }
replace => { "message" => "%{syslog_message}" }
remove_field => [ "syslog_message" ]
}
}
This part seems to be fine until I try to match a parsed field later in the filter. For example, like this:
if [syslog_program] =~ /nginx/ {
grok {
match => [ "message", "%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:agent}" ]
add_tag => [ "nginx" ]
}
}
I'm not really sure what to make of the error and googling hasn't helped much. I'm wondering if there is some issue or conflict with my timestamp?