When processing AWS CloudTrail logs, we noticed a discrepancy in the parsed AWS region compared to the region a log file originated. CloudTrail is not region specific , but monitors all configured regions and stuffs the collected log files into a s3 bucket. The s3 bucket of course is attached to a region, in our case eu-central-1.
We configured the AWS CloudTrail processor in Filebeat to listen to a sqs queue as suggested in the Filebeat configuration files to reproduce the problem from our production environment:
How to manual test this module
- Create a CloudTrail with a S3 bucket as the storage location
- Configure this S3 bucket to send "All object create events" to a SQS queue
- Configure filebeat, using the SQS queue url with s3 notification setup in
previous step.
This leads to the identical problematic situation, where the cloud.region is not identical to the region the log originated from, but the region where the s3 and sqs are located. For instance, "awsRegion":"us-east-1" is nonetheless parsed to eu-central-1, despite a correctly configured ingest pipeline:
{
"rename": {
"field": "json.awsRegion",
"target_field": "cloud.region",
"ignore_failure": true
}
}