Filebeat AWS Module Errors and Missing Logs

[johnnywalker (Jonathan Walker)]

Prior to our 7.8 migration, we used our own cloudtrail worker(golang + logstash) but wanted to adopt the aws filebeat module for ECS conversion.

Upon doing so I notice a significant drop in log volume(500GB -> 30GB), missing actions(ConsoleLogin), additional ECS fields(such as crowdstrike.event.StartTimestamp), and a significant amount of MissingRegion errors. Am I missing anything? Is there any additional setup requirements? Is the ingest pipeline not working? We are using ECK and just a standard filebeat deployment, not using the Beats resource kind, in kubernetes.

Any help would be greatly appreciated!

Numerous:
s3/input.go:206 SQS ReceiveMessageRequest failed: MissingRegion: could not find region configuration

Without S3 Input(filebeat.inputs):
s3/input.go:538 decode json failed for 'AWSLogs/000000000000/CloudTrail-Digest/us-east-1/2020/07/22/000000000000_CloudTrail-Digest_us-east-1_redacted-logging_us-east-1_20200722T030000Z.json.gz' from S3 bucket 'redacted-logging', skipping this file: json: cannot unmarshal string into Go value of type []interface {}

filebeat.yml:

setup.template.enabled: true
setup.ilm.enabled: false
setup.template.overwrite: true
setup.template.name: "cloudtrail"
setup.template.pattern: "cloudtrail-*"
setup.template.settings:
    index.number_of_shards: 3
    index.number_of_replicas: 1

filebeat.inputs:
- type: s3
  queue_url: "https://sqs.us-east-1.amazonaws.com/000000000000/logging-notifications"
  expand_event_list_from_field: Records

filebeat.modules:
- module: aws
  cloudtrail:
  enabled: true
  var.queue_url: https://sqs.us-east-1.amazonaws.com/000000000000/logging-notifications
  var.visibility_timeout: 300s
  elb.enabled: false
  s3access.enabled: false
  vpcflow.enabled: false


output.elasticsearch:
  hosts: ['https://elasticsearch.elastic-system.svc.cluster.local:9200']
  ssl.certificate_authorities: ["/usr/share/filebeat/ca.crt"]
  ssl.verification_mode: none
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}
  index: "cloudtrail-%{+yyyy.MM.dd}"
  pipeline: filebeat-7.8.1-aws-cloudtrail-pipeline