Filebeat AWS S3 Module not working

Elastic Stack Beats
1

Hi Team,
We are facing the filebeat aws s3 module issue in version 7.10 ,7.10.1, 7.11.0, 7.11.1, 7.11.2, 7.12.0
We have deployed the filebeat in k8s environment. And the ELK with ECK operator.

While we tried to use the filebeat aws s3 module to get the data SQS queue we are getting the below error.

ERROR:
2021-03-30T10:35:42.401Z ERROR [input.s3] s3/collector.go:107 SQS ReceiveMessageRequest failed: InvalidClientTokenId: The security token included in the request is invalid
status code: 403, request id: XXXXXX {"queue_url": "XXXXXXX", "region": "XXXXX"}

Note: We are not using the aws credentials we are using the IAM role to access the SQS and S3.

As we have verified we are all clear in IAM permission and we dont find any issue with IAM here.

Kindly provide the guides to overcome this issue ASAP

2

@Kaiyan_Sheng

3

Thanks for creating the issue here! Using role_arn still requires access key id and secret access key to be under the shared credential file [default] section. Here is the link for how to create the shared credentials file: Create a shared credentials file - Amazon Simple Email Service. This file by default is ~/.aws/credentials for Linux and macOS. Do you have this file with the correct credentials under [default] section?

4

Hy kaiyan,
Earlier we dont have above setup which u have mentioned but worked.

We used only role_arn

5

Hey @bapa I'm also looking at adding support for role_arn with given aws_access_key_id and aws_secret_access_key (without the shared credential file).

So only role_arn worked before but not working now?

6

Hi @Kaiyan_Sheng I also experienced similar issue with the latest Filebeat v7.12. It seems there was recent change that breaks IAM role usage attached to EC2 ( Fleet AWS Cloudtrail integration stops working after upgrading elastic agent to 7.12.0).

What you are suggesting (specify access key or credentials file) is valid for beats deployment outside AWS environment but that is against best practices for deployment within AWS (EC2 or ECS task).

See Configuring the AWS SDK for Go - AWS SDK for Go

AWS suggests to configure IAM permission in the following order of preference:

  1. Use IAM Task Role for ECS task
  2. Use IAM role attached to EC2 (technically it is a role associated to IAM instance profile that is attached to EC2)
  3. Use credentials file
  4. Use environment variables

With option 2 above it is really convenient and more secure because there's no access key to manage or to rotate, only need to attach a role to an instance.

AWS CLI and SDK supports all those options.

For Filebeat configuration I think Elastic should implement in such a way if none are configured (all blank) then it will fallback to attached Task Role or IAM instance role or introducing a boolean configuration for attached IAM Role.

1 Like
7

Thanks @nugroho-expereo for your suggestion!! We did add the var.role_arn config option in the aws module but right now it only works when credentials are set in the shared credential file under the default profile. Will investigate more here! Thank you again!!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.