Filebeat:aws:cloudtrail

Elastic Stack Beats
1

Hi all,

Had a question about the filebeat aws module - specifically the cloudtrail set.

I've managed to get this up and running, and as far as i understand, the process used for the above can be broken down into the following steps:

  1. cloudtrail event gets saved to S3
  2. S3 triggers a message to the SQS queue
  3. Filebeat gets this message, deletes it from the queue, grabs the file from s3, processes the file, and sends the output to kibana

However, my thoughts are that the action of Filebeat getting the message from SQS (and authenticating with AWS through IAM Role or User) will also produce more cloudtrail events. Hence won't we get into this infinite loop where the stages become:

  1. cloudtrail event gets saved to S3
  2. S3 triggers a message to the SQS queue
  3. Filebeat gets this message, deletes it from the queue, grabs the file from s3, processes the file, and sends the output to kibana
  4. Filebeat actions create new cloudtrail events
    Go back to 1) and run infinitely...

The issue then might become large bills on AWS usage for S3 and SQS...

Or am i missing something here?

2 Likes
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.