I have vpc flow logs going to an S3 bucket and an SQS notification for any object creation event.
I actually much preferred the logstash method of polling the bucket, for several reasons - mainly the ability to re-index from source easily, use regex matching of file patterns, and general simplicity. The realtime notifications don't provide me with much benefit and just add complexity. I hope to see polling as an option on filebeat some day.
Nonetheless, I've set it up, and I am using an instance role on my filebeat node for access to SQS and S3.
Although quite a bit of data is being read successfully and I can see it in Kibana, I'm getting a lot of errors and warnings. I have no easy way to tell if it's getting everything, but these messages imply that some may be failing and getting dropped. I can't be certain though, because they're very confusing messages.
2020-03-27T01:27:10.019Z ERROR [s3] s3/input.go:204 failed to receive message from SQS: MissingRegion: could not find region configuration
2020-03-27T01:27:20.019Z ERROR [s3] s3/input.go:204 failed to receive message from SQS: MissingRegion: could not find region configuration
2020-03-27T01:27:20.019Z ERROR [s3] s3/input.go:204 failed to receive message from SQS: MissingRegion: could not find region configuration
2020-03-27T01:27:30.020Z ERROR [s3] s3/input.go:204 failed to receive message from SQS: MissingRegion: could not find region configuration
2020-03-27T01:27:30.020Z ERROR [s3] s3/input.go:204 failed to receive message from SQS: MissingRegion: could not find region configuration
2020-03-27T01:27:40.020Z ERROR [s3] s3/input.go:204 failed to receive message from SQS: MissingRegion: could not find region configuration
2020-03-27T01:27:40.020Z ERROR [s3] s3/input.go:204 failed to receive message from SQS: MissingRegion: could not find region configuration
2020-03-27T01:27:40.887Z ERROR [s3] s3/input.go:475 ReadString failed: context deadline exceeded
2020-03-27T01:27:41.531Z ERROR [s3] s3/input.go:475 ReadString failed: context deadline exceeded
2020-03-27T01:27:41.531Z ERROR [s3] s3/input.go:386 createEventsFromS3Info failed for AWSLogs//vpcflowlogs/us-west-2/2020/03/26/_vpcflowlogs_us-west-2_fl-.log.gz: ReadString failed: conte
xt deadline exceeded
2020-03-27T01:27:41.571Z WARN [s3] s3/input.go:277 Processing message failed, updating visibility timeout
2020-03-27T01:27:41.636Z INFO [s3] s3/input.go:282 Message visibility timeout updated to 300
2020-03-27T01:27:41.794Z WARN [s3] s3/input.go:277 Processing message failed, updating visibility timeout
2020-03-27T01:27:41.799Z INFO [s3] s3/input.go:282 Message visibility timeout updated to 300
2020-03-27T01:27:41.993Z WARN [s3] s3/input.go:277 Processing message failed, updating visibility timeout
2020-03-27T01:27:41.997Z INFO [s3] s3/input.go:282 Message visibility timeout updated to 300
Hi @swisscheese, thank you for letting us know about your preference on polling method! I will create a github issue to track this.
For the error message you see in the log, I believe it's caused by other filesets that are enabled(by default) in aws module. We do have a github issue to fix this: https://github.com/elastic/beats/issues/17256
If you change aws.yml to the config below(assume you are running 7.6 version), you should see a better/cleaner log.
Ahh sorry I missed this error message. Question: seems like this error showed up twice for the same message AWSLogs/699536110035/vpcflowlogs/us-west-2/2020/03/29/_vpcflowlogs_us-west-2_fl-.log.gz in 1 second, which is not cool... Did you by any chance changed the visibility_timeout param?
Also since the error message is complaining about file AWSLogs/******/vpcflowlogs/eu-west-3/2020/04/09*******_vpcflowlogs_eu-west-3_fl-0629b5eccc8f3d0ad_20200409T0120Z_9b923cb5.log.gz, if you can check in Kibana discover (maybe with a filter) to see if there are events from this file please?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.