Parse AWS CloudTrail and CloudWatch Logs in Logstash

Elastic Stack Beats
1

By default, CloudTrail logs are aggregated per region and then redirected to an S3 bucket (compressed JSON files). Cloudtrail delivers log files to s3 bucket, approximately every 5 minutes. We can use the Logstash S3 input plugin or, alternatively, download the file and use the Logstash file input plugin. The compressed logs need to be de-compressed and then read -Custom code. (Please correct in case i am wrong)

I was going through the post for Community Beat, and found cloudwatchlogsbeat and cloudtrailbeat. DO they de-compress the CT and CW logs automatically from S3 or we need to write custom code for the same.

Any suggestions if Logstash S3 input plugin is better than these community beats?

1 Like
2

I am not familiar with the existing community Beats. However, Filebeat has a new s3 input: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-s3.html It is not able to parse, but you could use it to collect the logs and send it to a "central" LS or ES to parse those.

1 Like
3

We just opened a ticket to track the work for adding cloudtrail fileset in Filebeat: https://github.com/elastic/beats/issues/14657

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.