Hiya
I've upgraded to filebeat 7.9.0 and using the zeek (bro) module with its associated pipeline. I noticed a few entries in the dead letter queue, and a quick examination showed a parsing error on the date field of the zeek SMTP logs:
<LogStash::Event:0x12f05664>], response: {"index"=>{"_index"=>"filebeat-ops-7.9.0-000003", "_type"=>"_doc", "_id"=>"OU48cXQBDFnN035lvOEP", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [zeek.smtp.date] of type [date] in document with id 'OU48cXQBDFnN035lvOEP'. Preview of field's value: '9 Sep 2020 05:00:05 +0000'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [9 Sep 2020 05:00:05 +0000] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"}}}}
I had a quick look at the zeek-smtp-pipeline and found that the data parser was limited to:
{
"field": "zeek.smtp.date",
"target_field": "zeek.smtp.date",
"formats": [
"EEE, d MMM yyyy HH:mm:ss Z"
],
"if": "ctx.zeek.smtp.date != null"
}
It didn't seem to like the date (9 Sep 2020 05:00:05 +0000) so I added a couple of extra date lines like such, and the issue seems to have gone away:
{
"field": "zeek.smtp.date",
"target_field": "zeek.smtp.date",
"formats": [
"EEE, d MMM yyyy HH:mm:ss Z",
"d MMM yyyy HH:mm:ss Z",
"dd MMM yyyy HH:mm:ss Z"
],
"if": "ctx.zeek.smtp.date != null"
}