Input 'aws-s3' failed with: failed to initialize s3 poller

Its failing to find the region for ur bucket. Thats where its erroring. Are you sure the credentials are correct and have the right IAM permissions?

@legoguy1000 yes, the credentials are correct and have permissions since the same IAM user is being used by ES for snapshot in the same bucket.

Just to confirm, it has at min the below? Ref AWS S3 input | Filebeat Reference [7.15] | Elastic

s3:GetObject
s3:ListBucket
s3:GetBucketLocation

@legoguy1000 yes, all the permissions specified are allowed for the IAM user.

OK.

The only other thing thats weird is that AWS doesn't support path style buckets so this should be likehttps://test-bucket-1-stos.s3.us-east-1.amazonaws.com/?location

Or possibly per the docs To use this implementation of the operation, you must be the bucket owner. Are the credentials you're using associated with the account that owns the bucket? At this point i'm running out of ideas.

@Ayush_Mathur

can you try running the following aws-cli command using the IAM resource used by filebeat?

aws s3api get-bucket-location --bucket test-bucket-1-stos

As for the config params:
endpoint is not needed if your bucket must not be accessed from a custom domain, like your case
default_region is not supported in filebeat. You can set the env variable AWS_DEFAULT_REGION to override it if you want.

If not region is set (either missing or empty AWS_DEFAULT_REGION env or setting in ~/.aws/config or in EC2 instance metadata) us-east-1 is used as fallback.

This will be the region used for the GetBucketLocation request, once we get the region of the bucket we will use it in the following AWS call to that bucket.

GetBucketLocation is not region bound: you can perform the request from any region to a bucket in any other regions, and receive a proper result.

1 Like

The call is on the s3 api endpoint, not on the bucket endpoint, that's why the url called is https://s3.us-east-1.amazonaws.com/test-bucket-1-stos?location=

Ok, I was going off of GetBucketLocation - Amazon Simple Storage Service which shows virtual host style.

1 Like

@legoguy1000
Apparently you can call with any combination of bucket being in the host or the path and it will work anyway. If you have different buckets between host and path, the result will be for the host one

3 Likes

I tried configuring the module all over again but tried using IAM role which was used to create the bucket. Following are the 3 different erroneous responses I received:

  1. Only IAM role in input and module:
{"level":"info","timestamp":"2021-11-26T07:10:41.506Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":"861367A1E96D11A8"}
{"level":"info","timestamp":"2021-11-26T07:10:41.507Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":"3B61DF7D34A8DA0A"}
{"level":"error","timestamp":"2021-11-26T07:10:51.509Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: EC2RoleRequestError: no EC2 instance role found\ncaused by: request canceled, context deadline exceeded","id":"861367A1E96D11A8"}
{"level":"error","timestamp":"2021-11-26T07:10:51.509Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: EC2RoleRequestError: no EC2 instance role found\ncaused by: request canceled, context deadline exceeded","id":"3B61DF7D34A8DA0A"}
  1. Both IAM role and IAM User:
{"level":"info","timestamp":"2021-11-26T07:13:25.530Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":" D3FFBA17E3C353A"}
{"level":"info","timestamp":"2021-11-26T07:13:25.534Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":"B04D437DA3F9FE98"}
{"level":"error","timestamp":"2021-11-26T07:13:27.969Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Post \"https://sts.us-east-1.amazonaws.com/\": Forbidden","id":"B04D437DA3F9FE98"}
{"level":"error","timestamp":"2021-11-26T07:13:29.713Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Post \"https://sts.us-east-1.amazonaws.com/\": Forbidden","id":" D3FFBA17E3C353A"}
  1. Added sts endpoint to NO_PROXY env var of filebeat:
{"level":"info","timestamp":"2021-11-26T07:15:40.914Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":" D3FFBA17E3C353A"}
{"level":"info","timestamp":"2021-11-26T07:15:40.920Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":"B04D437DA3F9FE98"}
{"level":"error","timestamp":"2021-11-26T07:17:11.695Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Post \"https://sts.us-east-1.amazonaws.com/\": dial tcp 54.239.16.72:443: i/o timeout","id":"B04D437DA3F9FE98"}
{"level":"error","timestamp":"2021-11-26T07:17:13.811Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Post \"https://sts.us-east-1.amazonaws.com/\": dial tcp 54.239.16.72:443: i/o timeout","id":" D3FFBA17E3C353A"}

It seems the documentation for 7.15.0 is bit off as it says to either use IAM user keys OR session token OR IAM role. However, just using IAM role is not working in this case.

@Ayush_Mathur

can you share your module config?
did you attach the role to the ec2 instance in case 1?

@Andrea_Spacca no, I didn't attach anything to my EC2 since it was nowhere mentioned in documentation: AWS module | Filebeat Reference [7.15] | Elastic

@Ayush_Mathur
if the EC2 instance has no attached the role defined in the module attached it will use the AssumeRole API how it is described by the documentation: AWS module | Filebeat Reference [7.15] | Elastic

Did you proceed with step 3 and 4?

Hello @Andrea_Spacca ,
I tried that AWS CLI command you mentioned above using the IAM user (which is also used by ES for snapshot storage in S3 bucket). The output is attached as image here.
image

The same IAM user, when used in filebeat is failing to poll s3 and give following error:

{"level":"error","timestamp":"2021-11-30T08:25:14.735Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.amazonaws.com/es-backup-xxxx?location=\": Forbidden","id":"EB4DF85955C0CAB1"}

where filebeat is running on same host as Elasticsearch node.

@Ayush_Mathur

Could you please share your module config? (anonymised)
Could you also enable debug logs in Filebeat and post the whole output?

Did you set an endpoint in the config?

@Andrea_Spacca I can't really switch on the debug mode since beats is running on 50 nodes as daemon set. Is there any way to test the connection from pod terminal in verbose mode ?
The config is as follows:

  - module: aws
    cloudtrail:
      enabled: false
      var.bucket_arn: 'arn:aws:s3:::es-backup-xxxxx'
      var.bucket_list_interval: 300s
      var.number_of_workers: 5
      var.access_key_id: ${ACCESS_KEY}
      var.secret_access_key: ${SECRET_KEY}
      var.visibility_timeout: 300s
      var.api_timeout: 120s
      var.endpoint: amazonaws.com

also mentioned initially when writing this post :slight_smile: (just the bucket has changed)

@Ayush_Mathur

please, remove var.endpoint. it is not really needed since you are using the default one and this is causing to build the host for the get bucket request as s3.us-east-1.amazonaws.com

I suppose the host machine where Filebeat runs is not in us-east-1, am I correct?
This setup from aws cli generates an initial body response like the following:
<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region \'eu-central-1\' is wrong; expecting \'us-east-1\'</Message><Region>us-east-1</Region>

aws cli handles the error and does a proper follow up request. My assumption is that the AWS SDK is not so smart.

@Ayush_Mathur
you should be able to run /bin/sh on the pod with kubectl exec
once you are logged in the pod you can test the two following aws cli command:

aws s3api get-bucket-location --debug --bucket es-backup-xxxxx --endpoint-url https://s3.us-east-1.amazonaws.com

and

aws s3api get-bucket-location --debug --bucket es-backup-xxxxx

@Andrea_Spacca removing the endpoint did't make any difference, the error is still present as Forbidden:

{"level":"error","timestamp":"2021-11-30T10:29:46.175Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.amazonaws.com/es-backup-xxxxxx?location=\": Forbidden","id":"563F408588050113"}

The host machine is in eu-west-1 region.
Also, filebeat doesn't know of aws command, is there some specific plugin I need to install in filebeat ?

@Ayush_Mathur

Filebeat image is based on CentOS, you should be able to install aws cli with yum

I tried to reproduce Filebeat 7.15.0 enabling AWS SDK requests debugging and I see that the host set is indeed s3.us-east-1.amazonaws.com, and the GetBucketLocation is returning correct response from an host machine in eu-central-1

I would look in the permission of the role and bucket policy, in case there is any constraint on the region where the request comes from.