Ingesting CloudTrail logs from multiple AWS accounts

[vishakh]

Brief: We do have multiple AWS accounts and we would prefer to ingest cloudtrail logs from all our accounts via Filebeat's AWS Module. (filebeat from same/single server & single agent)
ELK-Stack: 7.8.0
Filebeat: 7.8.0
OS: AmazonLinux-2

While going through the Filebeat's AWS Module:

- module: aws
  cloudtrail:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.shared_credential_file: /etc/filebeat/aws_credentials
    #var.credential_profile_name: fb-aws
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token
    #var.visibility_timeout: 300s
    #var.api_timeout: 120s
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

From the above config snippet I don't see a way to configure multiple accounts, I would request a help on how to ingest/configure cloudtrail logs from multiple aws accounts via filebeat.

[Kaiyan_Sheng]

Hello! Thanks for posting your question here. For running Filebeat with multiple AWS accounts, you can just duplicate this config section with different sets of credentials. For example:

- module: aws
  cloudtrail:
    enabled: true
    var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    var.shared_credential_file: /etc/filebeat/aws_credentials_for_account_1
- module: aws
  cloudtrail:
    enabled: true
    var.queue_url: https://sqs.myregion.amazonaws.com/abcde/myqueue
    var.shared_credential_file: /etc/filebeat/aws_credentials_for_account_2

[vishakh]

@Kaiyan_Sheng Thanks for the reply and it was really helpful
I just an adjoining question:
Here in the below filebeat config, I'm using cloudtrail file-set module and since I'm specifically using 'cloudtrail' module do I still need to supply sqs path?

- module: aws
  cloudtrail:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

Here, I'll passing the aws keys but along these should I pass/add SQS queue URL?
Can't the 'Cloudtrail module' pull the Cloudtrail Logs directly without using SQS?

[Kaiyan_Sheng]

Hey @vishakh, unfortunately you still need to setup SQS for Cloudtrail logs. Cloudtrail logs need to go into S3 bucket and the S3 bucket needs to setup with SQS for notification. https://www.elastic.co/blog/getting-aws-logs-from-s3-using-filebeat-and-the-elastic-stack This might help for the setup.