I had set up filebeat agent in my AWS account to ship logs from an S3 bucket. The s3 bucket contains Cloudtrail logs and VPC Flow Logs and has Access logging enabled.
The file 'filebeat/modules.d/aws.yml' was modified to enable the filesets Cloudtrail, Cloudwatch, VPCFlowlogs and s3access and add the corresponding SQS queue ARN along with the required credentials.
The data is flowing in and is being successfully received from Filebeat AWS module in Elasticsearch. However, the data is incorrectly mapped in the logs.
I can see vpc flow logs associated with Cloudwatch dataset and Cloudtrail datasets, similarly, Cloudtrail logs are being tagged as s3access fileset (Please refer to the attached image).
Please help understand how the data is selected by Filebeat filesets and how the mapping can be corrected.
Have you tried the AWS specific dashboards under the Dashboards tab on the left? That might show you something more specifically. All filters you want can be applied both on your screen and in each dashboard
Yes Marius. I have explored the Dashboards but didn't find anything significant that could help. The issue is with the incorrect mapping between data and filesets.
Thank you!
All the logs are stored in one centralised S3 bucket and have the same queue set to receive event notifications on new objects ingestion in the bucket.
Does filebeat requires separate buckets and queues for data from different services?
Because my use case requires me to have all the logs in a single bucket.
Because they are separate filesets each fileset is expecting only to receive its specific logs so i suspect you need different buckets/queues. Are you seeing the same log/event show up multiple times as part of the different filesets?
No Alex, the logs are not repeated as part of different filesets.
Do you think I can find any documentation on this for more clarity on the requirement of separate buckets and queues?
That could be a problem, given my use case
I'm not necessarily seeing anything in the docs saying they have to be separate. Are u using the same queue for all the filesets? If so that is probably the issue. Idk if u can create separate queues for a single bucket for different data.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
