Prior to our 7.8 migration, we used our own cloudtrail worker(golang + logstash) but wanted to adopt the aws filebeat module for ECS conversion.
Upon doing so I notice a significant drop in log volume(500GB -> 30GB), missing actions(ConsoleLogin), additional ECS fields(such as crowdstrike.event.StartTimestamp), and a significant amount of MissingRegion errors. Am I missing anything? Is there any additional setup requirements? Is the ingest pipeline not working? We are using ECK and just a standard filebeat deployment, not using the Beats resource kind, in kubernetes.
Any help would be greatly appreciated!
Numerous:
s3/input.go:206 SQS ReceiveMessageRequest failed: MissingRegion: could not find region configuration
Without S3 Input(filebeat.inputs):
s3/input.go:538 decode json failed for 'AWSLogs/000000000000/CloudTrail-Digest/us-east-1/2020/07/22/000000000000_CloudTrail-Digest_us-east-1_redacted-logging_us-east-1_20200722T030000Z.json.gz' from S3 bucket 'redacted-logging', skipping this file: json: cannot unmarshal string into Go value of type []interface {}
filebeat.yml:
setup.template.enabled: true
setup.ilm.enabled: false
setup.template.overwrite: true
setup.template.name: "cloudtrail"
setup.template.pattern: "cloudtrail-*"
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 1
filebeat.inputs:
- type: s3
queue_url: "https://sqs.us-east-1.amazonaws.com/000000000000/logging-notifications"
expand_event_list_from_field: Records
filebeat.modules:
- module: aws
cloudtrail:
enabled: true
var.queue_url: https://sqs.us-east-1.amazonaws.com/000000000000/logging-notifications
var.visibility_timeout: 300s
elb.enabled: false
s3access.enabled: false
vpcflow.enabled: false
output.elasticsearch:
hosts: ['https://elasticsearch.elastic-system.svc.cluster.local:9200']
ssl.certificate_authorities: ["/usr/share/filebeat/ca.crt"]
ssl.verification_mode: none
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
index: "cloudtrail-%{+yyyy.MM.dd}"
pipeline: filebeat-7.8.1-aws-cloudtrail-pipeline