7.9.2 FileBeat CloudTrail AWS Module Changes Break Log Collection

[djcullen]

It appears that the changes made to the FileBeat AWS module in 7.9.2 (specifically 21086) break the module for anyone whose CloudTrail S3 object key does not conform to the following regex:
^AWSLogs/\d+/CloudTrail/

My organization's CloudTrail logs follow the naming convention below which doesn't conform to the baked-in regex above and results in FileBeat simply deleting all messages in my SQS queue when using the AWS module:
mgmt/AWSLogs/8**********/CloudTrail/

I can disable the AWS module and declare the following regex using file_selectors in my filebeat.yml, but I have not been able to do so successfully within aws.yml (I've tried creating a var.file_selectors parameter, but it is just ignored):
mgmt\/AWSLogs\/\d+\/CloudTrail\/.*

I only want to collect the mgmt logs and not the data or digest logs, and this regex works for that purpose. I'd prefer to use the AWS module. Is there something I'm missing that will allow me to change the default regex used in the AWS module?

[mtojek (Marcin Tojek)]

I'm afraid you won't be able to fix it without using a custom config based on the S3 input (and file selectors).

What you can do is to manually adjust the ingest pipeline in Elasticsearch.

[djcullen]

Hi Marcin. I was hoping for a more efficient solution to this issue, but I appreciate the response and workaround. I'll keep the AWS module disabled and hope for some flexibility down the road. Thanks!