I'm trying to configure Filebeat for the first time with an S3 input but so far I've been unsuccessful in getting Filebeat set up correctly. It seems to read messages from SQS and S3, but when it comes time to delete the message from SQS I get this error:
2020-04-28T04:43:45.369Z WARN [cfgwarn] s3/input.go:129 BETA: s3 input type is used
2020-04-28T04:43:45.370Z INFO input/input.go:114 Starting input of type: s3; ID: 10210413665917225791
2020-04-28T04:43:45.370Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2020-04-28T04:43:45.370Z INFO [s3] s3/input.go:174 visibility timeout is set to 300 seconds
2020-04-28T04:43:45.371Z INFO [s3] s3/input.go:175 aws api timeout is set to 2m0s
2020-04-28T04:43:45.371Z INFO [s3] s3/input.go:196 s3 input worker has started. with queueURL: https://sqs.ca-central-1.amazonaws.com/MYACCOUNTNUMBER/QUEUENAMEREDACTED
2020-04-28T04:43:57.109Z WARN [s3] s3/input.go:521 Decode json failed for 'REDACTED.json', skipping this file
2020-04-28T04:43:57.203Z ERROR [s3] s3/input.go:291 deleteMessages failed: DeleteMessageRequest failed: AccessDenied: Access to the resource https://sqs.ca-central-1.amazonaws.com/ is denied.
status code: 403, request id: 3c05b30f-ccb3-59c0-8ae8-e42e7a7e5f5e
My Filebeat configuration is very straightforward:
filebeat.inputs:
- type: s3
queue_url: https://sqs.ca-central-1.amazonaws.com/MYACCOUNTNUMBER/QUEUENAMEREDACTED
role_arn: ROLENAMEREDACTED
expand_event_list_from_field: Records
And I have an IAM Role that is being assumed by filebeat. The IAM role has a policy like this:
"Statement": [
{
"Sid": "ReadFromSQS",
"Effect": "Allow",
"Action": [
"sqs:ReceiveMessage",
"sqs:GetQueueUrl",
"sqs:GetQueueAttributes",
"sqs:DeleteMessageBatch",
"sqs:DeleteMessage",
"sqs:ChangeMessageVisibilityBatch",
"sqs:ChangeMessageVisibility",
"sqs:*"
],
"Resource": "arn:aws:sqs:ca-central-1:MYACCOUNT:QUEUENAMEREDACTED"
},
I am using Filebeat 7.6.2.
Furthermore, attempting the same operations using aws-cli results in no error.
aws --region ca-central-1 --profile filebeat sqs receive-message --queue-url 'https://sqs.ca-central-1.amazonaws.com/MYACCOUNT/REDACTEDQUEUENAME'
aws --region ca-central-1 --profile filebeat sqs delete-message --queue-url 'https://sqs.ca-central-1.amazonaws.com/MYACCOUNT/QUEUENAMEREDACTED' --receipt-handle 'AVERYLONGSTRING'