Filebeat Zeek and date parsing with SMTP broken

Hiya

I've upgraded to filebeat 7.9.0 and using the zeek (bro) module with its associated pipeline. I noticed a few entries in the dead letter queue, and a quick examination showed a parsing error on the date field of the zeek SMTP logs:

<LogStash::Event:0x12f05664>], response: {"index"=>{"_index"=>"filebeat-ops-7.9.0-000003", "_type"=>"_doc", "_id"=>"OU48cXQBDFnN035lvOEP", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [zeek.smtp.date] of type [date] in document with id 'OU48cXQBDFnN035lvOEP'. Preview of field's value: '9 Sep 2020 05:00:05 +0000'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [9 Sep 2020 05:00:05 +0000] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"}}}}
I had a quick look at the zeek-smtp-pipeline and found that the data parser was limited to:

{
   "field": "zeek.smtp.date",
   "target_field": "zeek.smtp.date",
   "formats": [
     "EEE, d MMM yyyy HH:mm:ss Z"    
   ],
   "if": "ctx.zeek.smtp.date != null"
 }

It didn't seem to like the date (9 Sep 2020 05:00:05 +0000) so I added a couple of extra date lines like such, and the issue seems to have gone away:

{
  "field": "zeek.smtp.date",
  "target_field": "zeek.smtp.date",
  "formats": [
    "EEE, d MMM yyyy HH:mm:ss Z",
    "d MMM yyyy HH:mm:ss Z",
    "dd MMM yyyy HH:mm:ss Z"
  ],
  "if": "ctx.zeek.smtp.date != null"
}

hi @rossw, I do not see any major changes in the pipeline in 7.9, but since you have found a solution for your case, feel free to create an issue/PR to the beats repo describing the scenario and the changes in the pipeline.