The documentation for the AWS module states "It uses filebeat s3 input to get log files from AWS S3 buckets with SQS notification or directly polling list of S3 objects in an S3 bucket. The use of SQS notification is preferred: polling list of S3 objects is expensive in terms of performance and costs, and cannot scale horizontally without ingestion duplication, and should be preferably used only when no SQS notification can be attached to the S3 buckets.". I have been working to find a way to ingest existing log data, which the SQS queue mechanism does not support since it only polls the logs that are created after the time of the queue creation. My question is there other ways to ingest all the log contents in a Cloudtrail log bucket without using the SQS queue?.
Thanks in Advance!
Are you asking if u can ingest from S3 without SQS and without polling the bucket?
I am asking how we can ingest the whole cloud trail bucket content so that I will be able to get the logs that were generated prior to the time we started ingesting.
Ok. We had a similar situation where we needed to ingest historical logs from the bucket. We just downloaded them, had a script that extracted the json and consolidated it into another file. Then we used the filebeat AWS modules and changed the input to the log input instead of the aws-s3 input.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.