Can logstash log event when parsing error?

Elastic Stack Logstash
1

Hi all,

To explain my question, assume the following scenario:

I use logstash to handle huge logs, and I notice that /var/log/logstash/logstash.log has an parsing error message:

{:timestamp=>"2016-03-25T10:00:21.365000", :message=>"Failed parsing date from field", :field=>"Timestamp", :value=>"727;%20.NET%20CLR%203.5.30729;%20.NET%20CLR%203.0.30729;%20Media%20Center%20PC%206.0;%20.NET4.0E;%20.NET4.0C)", :exception=>"Invalid format: \"727;%20.NET%20CLR%203.5.30729;%20.N...\" is malformed at \";%20.NET%20CLR%203.5.30729;%20.N...\"", :config_parsers=>"YYYYMMddHHmmss", :config_locale=>"default=en_US", :level=>:warn}

But I cannot figure out what's wrong with my log from above message. If Iogstash can log full event content when error occurred, then I can figure out the problem or do more tests.

So, my question is: can logstash record content of event when parsing error? Or there is any better approach to debug parsing error in production environment? Thanks.

2

When the date filter can't parse a field it'll add the _dateparsefailure tag to the event. You can emit those messages to a separate logfile or to Logstash's regular log, e.g. like this:

output {
  if "_dateparsefailure" in [tags] {
    file {
      path => "/var/log/logstash/dateparsefailure.log"
      codec => json_lines
    }
  }
}
1 Like