Ingesting CloudTrail logs from multiple AWS accounts

Brief: We do have multiple AWS accounts and we would prefer to ingest cloudtrail logs from all our accounts via Filebeat's AWS Module. (filebeat from same/single server & single agent)
ELK-Stack: 7.8.0
Filebeat: 7.8.0
OS: AmazonLinux-2

While going through the Filebeat's AWS Module:

- module: aws
  cloudtrail:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.shared_credential_file: /etc/filebeat/aws_credentials
    #var.credential_profile_name: fb-aws
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token
    #var.visibility_timeout: 300s
    #var.api_timeout: 120s
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

From the above config snippet I don't see a way to configure multiple accounts, I would request a help on how to ingest/configure cloudtrail logs from multiple aws accounts via filebeat.

Hello! Thanks for posting your question here. For running Filebeat with multiple AWS accounts, you can just duplicate this config section with different sets of credentials. For example:

- module: aws
  cloudtrail:
    enabled: true
    var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    var.shared_credential_file: /etc/filebeat/aws_credentials_for_account_1
- module: aws
  cloudtrail:
    enabled: true
    var.queue_url: https://sqs.myregion.amazonaws.com/abcde/myqueue
    var.shared_credential_file: /etc/filebeat/aws_credentials_for_account_2
1 Like

@Kaiyan_Sheng Thanks for the reply and it was really helpful
I just an adjoining question:
Here in the below filebeat config, I'm using cloudtrail file-set module and since I'm specifically using 'cloudtrail' module do I still need to supply sqs path?

- module: aws
  cloudtrail:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

Here, I'll passing the aws keys but along these should I pass/add SQS queue URL?
Can't the 'Cloudtrail module' pull the Cloudtrail Logs directly without using SQS?

Hey @vishakh, unfortunately you still need to setup SQS for Cloudtrail logs. Cloudtrail logs need to go into S3 bucket and the S3 bucket needs to setup with SQS for notification. https://www.elastic.co/blog/getting-aws-logs-from-s3-using-filebeat-and-the-elastic-stack This might help for the setup.