SQS DeleteMessageRequest failed with S3 input in Filebeat

[nhnicwaller (Nic Waller)]

I'm trying to configure Filebeat for the first time with an S3 input but so far I've been unsuccessful in getting Filebeat set up correctly. It seems to read messages from SQS and S3, but when it comes time to delete the message from SQS I get this error:

2020-04-28T04:43:45.369Z        WARN    [cfgwarn]       s3/input.go:129 BETA: s3 input type is used
2020-04-28T04:43:45.370Z        INFO    input/input.go:114      Starting input of type: s3; ID: 10210413665917225791 
2020-04-28T04:43:45.370Z        INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1
2020-04-28T04:43:45.370Z        INFO    [s3]    s3/input.go:174 visibility timeout is set to 300 seconds
2020-04-28T04:43:45.371Z        INFO    [s3]    s3/input.go:175 aws api timeout is set to 2m0s
2020-04-28T04:43:45.371Z        INFO    [s3]    s3/input.go:196 s3 input worker has started. with queueURL: https://sqs.ca-central-1.amazonaws.com/MYACCOUNTNUMBER/QUEUENAMEREDACTED
2020-04-28T04:43:57.109Z        WARN    [s3]    s3/input.go:521 Decode json failed for 'REDACTED.json', skipping this file
2020-04-28T04:43:57.203Z        ERROR   [s3]    s3/input.go:291 deleteMessages failed: DeleteMessageRequest failed: AccessDenied: Access to the resource https://sqs.ca-central-1.amazonaws.com/ is denied.
        status code: 403, request id: 3c05b30f-ccb3-59c0-8ae8-e42e7a7e5f5e

My Filebeat configuration is very straightforward:

filebeat.inputs:
  - type: s3
    queue_url: https://sqs.ca-central-1.amazonaws.com/MYACCOUNTNUMBER/QUEUENAMEREDACTED
    role_arn: ROLENAMEREDACTED
    expand_event_list_from_field: Records

And I have an IAM Role that is being assumed by filebeat. The IAM role has a policy like this:

"Statement": [
    {
        "Sid": "ReadFromSQS",
        "Effect": "Allow",
        "Action": [
            "sqs:ReceiveMessage",
            "sqs:GetQueueUrl",
            "sqs:GetQueueAttributes",
            "sqs:DeleteMessageBatch",
            "sqs:DeleteMessage",
            "sqs:ChangeMessageVisibilityBatch",
            "sqs:ChangeMessageVisibility",
            "sqs:*"
        ],
        "Resource": "arn:aws:sqs:ca-central-1:MYACCOUNT:QUEUENAMEREDACTED"
    },

I am using Filebeat 7.6.2.

Furthermore, attempting the same operations using aws-cli results in no error.

aws --region ca-central-1 --profile filebeat sqs receive-message --queue-url 'https://sqs.ca-central-1.amazonaws.com/MYACCOUNT/REDACTEDQUEUENAME'

aws --region ca-central-1 --profile filebeat sqs delete-message --queue-url 'https://sqs.ca-central-1.amazonaws.com/MYACCOUNT/QUEUENAMEREDACTED' --receipt-handle 'AVERYLONGSTRING'

[nhnicwaller (Nic Waller)]

My problem was that role_arn is not supported yet in the version of Filebeat I'm using (7.6.2). It will be available in the 7.8 release.

So I thought I was granting access with the IAM role, but actually a smaller set of limited rights was being granted from somewhere else.