Input 'aws-s3' failed with: failed to initialize s3 poller

Elastic Stack Beats
1

Hi,

I've been trying to setup AWS module on 7.15 Elastic Stack cluster running as containers to ingest cloudtrail, cloudwatch, elb, s3access and vpcflow events (configured only for cloudtrail atm).
However, there seems to be some bug in filebeat wherein when bucket.arn is configured, it always appends us-east-1 region by default making the S3 bucket inaccessible.

The relevant filebeat config is as follows:

  - module: aws
    cloudtrail:
      enabled: true
      var.bucket_arn: 'arn:aws:s3:::test-bucket'
      var.bucket_list_interval: 300s
      var.number_of_workers: 5
      var.access_key_id: ${ACCESS_KEY}
      var.secret_access_key: ${SECRET_KEY}
      var.visibility_timeout: 300s
      var.api_timeout: 120s
      var.endpoint: amazonaws.com
      var.default_region: "${AWS_REGION:eu-west-1}"

with S3 type input set as:

- type: aws-s3
  bucket_arn: arn:aws:s3:::test-bucket
  number_of_workers: 5
  bucket_list_interval: 300s
  expand_event_list_from_field: Records
  access_key_id: ${ACCESS_KEY}
  secret_access_key: ${SECRET_KEY}
  endpoint: amazonaws.com
  default_region: "${AWS_REGION:eu-west-1}"
  parsers:
    - multiline:
        pattern: "^<Event"
        negate:  true
        match:   after

However, no matter what config I provide (when using bucket_arn), us-east-1 is always appended to the connection string. Sample logs are as follows:

{"level":"error","timestamp":"2021-10-29T09:32:18.382Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.eu-west-1.amazonaws.com/test-bucket?location=\": Forbidden","id":"7C946250A2CBE8D6"}

{"level":"error","timestamp":"2021-10-29T09:42:04.606Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.s3.eu-west-1.amazonaws.com/test-bucket?location=\": x509: certificate is valid for s3-eu-west-1.amazonaws.com, *.s3-eu-west-1.amazonaws.com, s3.eu-west-1.amazonaws.com, *.s3.eu-west-1.amazonaws.com, s3.dualstack.eu-west-1.amazonaws.com, *.s3.dualstack.eu-west-1.amazonaws.com, *.s3.amazonaws.com, *.s3-control.eu-west-1.amazonaws.com, s3-control.eu-west-1.amazonaws.com, *.s3-control.dualstack.eu-west-1.amazonaws.com, s3-control.dualstack.eu-west-1.amazonaws.com, *.s3-accesspoint.eu-west-1.amazonaws.com, *.s3-accesspoint.dualstack.eu-west-1.amazonaws.com, *.s3.eu-west-1.vpce.amazonaws.com, not s3.us-east-1.s3.eu-west-1.amazonaws.com","id":"25BD02C5CFA1AFB3"}

{"level":"error","timestamp":"2021-10-29T09:49:33.194Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.https://s3.eu-west-1.amazonaws.com/test-bucket/test-bucket?location=\": Forbidden","id":"73D457479F8D5CDB"}

{"level":"error","timestamp":"2021-10-29T10:00:02.343Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.amazonaws.com/test-bucket?location=\": Forbidden","id":"BDCC88C015F87747"}

@Kaiyan_Sheng can you please help me to resolve this, may be I'm missing something ?

2

I don't see anywhere in the code where default_region config is a thing. Have u tried not setting it? You shouldn't need to, it will determine the region where the S3 bucket exists. How did u generate those errors, what were the configs for each?

3

@legoguy1000 I added default_region to try if that makes filebeat to use my region instead of default us-east-1 (took it from metricbeat configuration where it works fine).
As far as those errors are concerned, they are:

  1. without specifying default_region
  2. specifying default_region
  3. adding full s3 bucket endpoint with https (duplicate https in url)
  4. adding full s3 bucket endpoint without https (duplicate s3.*** in url)
4

Also, I don't see any documentation regarding SSL params to be used if my proxy requires them. Any idea what parameter should be used in this case ?

5

So if u specify a fully url as the endpoint it will not work in 7.15. As for default_region, that exists for metricbeat but has been deprecated and removed in 8.0.0 but I couldn't find anything for Filebeat. I'd remove it and try again. The aws-s3 input will query the us-east-1 endpoint to determine where the S3 bucket is and then when it actually polls it, will use the appropriate region endpoint for where the bucket is located. Also u don't need to specify the endpoint config if it's amazonaws.com. As for the proxy config, there was a bug that @kaiyan-sheng found, Change proxy_url from url.URL to string by kaiyan-sheng · Pull Request #28725 · elastic/beats · GitHub, so it may not work. But currently there are no SSL configs for the aws-3 proxy url.

6

@legoguy1000 I tried removing the default_region as per your suggestion but kept endpoint config. The result is still the same:

{"level":"info","timestamp":"2021-11-02T16:04:11.250Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":"2000C88E6AA7EA04"}
{"level":"info","timestamp":"2021-11-02T16:04:11.253Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":"  BE301CC83DBCE5"}
{"level":"error","timestamp":"2021-11-02T16:04:12.169Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.amazonaws.com/test-bucket-1-stos?location=\": Forbidden","id":"  BE301CC83DBCE5"}
{"level":"error","timestamp":"2021-11-02T16:04:13.377Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.amazonaws.com/test-bucket-1-stos?location=\": Forbidden","id":"2000C88E6AA7EA04"}

So I can infer two things here:

  1. I may need proxy_url to be configured (but it needs SSL which is not yet available)
  2. I may need to switch to SQS queue instead which will require whole new level of TA analysis since it's not yet in approved list of services in our project.
7

Its failing to find the region for ur bucket. Thats where its erroring. Are you sure the credentials are correct and have the right IAM permissions?

8

@legoguy1000 yes, the credentials are correct and have permissions since the same IAM user is being used by ES for snapshot in the same bucket.

9

Just to confirm, it has at min the below? Ref AWS S3 input | Filebeat Reference [7.15] | Elastic

s3:GetObject
s3:ListBucket
s3:GetBucketLocation
10

@legoguy1000 yes, all the permissions specified are allowed for the IAM user.

11

OK.

The only other thing thats weird is that AWS doesn't support path style buckets so this should be likehttps://test-bucket-1-stos.s3.us-east-1.amazonaws.com/?location

Or possibly per the docs To use this implementation of the operation, you must be the bucket owner. Are the credentials you're using associated with the account that owns the bucket? At this point i'm running out of ideas.

12

@Ayush_Mathur

can you try running the following aws-cli command using the IAM resource used by filebeat?

aws s3api get-bucket-location --bucket test-bucket-1-stos

As for the config params:
endpoint is not needed if your bucket must not be accessed from a custom domain, like your case
default_region is not supported in filebeat. You can set the env variable AWS_DEFAULT_REGION to override it if you want.

If not region is set (either missing or empty AWS_DEFAULT_REGION env or setting in ~/.aws/config or in EC2 instance metadata) us-east-1 is used as fallback.

This will be the region used for the GetBucketLocation request, once we get the region of the bucket we will use it in the following AWS call to that bucket.

GetBucketLocation is not region bound: you can perform the request from any region to a bucket in any other regions, and receive a proper result.

1 Like
13

The call is on the s3 api endpoint, not on the bucket endpoint, that's why the url called is https://s3.us-east-1.amazonaws.com/test-bucket-1-stos?location=

14

Ok, I was going off of GetBucketLocation - Amazon Simple Storage Service which shows virtual host style.

1 Like
15

@legoguy1000
Apparently you can call with any combination of bucket being in the host or the path and it will work anyway. If you have different buckets between host and path, the result will be for the host one

3 Likes
16

I tried configuring the module all over again but tried using IAM role which was used to create the bucket. Following are the 3 different erroneous responses I received:

  1. Only IAM role in input and module:
{"level":"info","timestamp":"2021-11-26T07:10:41.506Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":"861367A1E96D11A8"}
{"level":"info","timestamp":"2021-11-26T07:10:41.507Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":"3B61DF7D34A8DA0A"}
{"level":"error","timestamp":"2021-11-26T07:10:51.509Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: EC2RoleRequestError: no EC2 instance role found\ncaused by: request canceled, context deadline exceeded","id":"861367A1E96D11A8"}
{"level":"error","timestamp":"2021-11-26T07:10:51.509Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: EC2RoleRequestError: no EC2 instance role found\ncaused by: request canceled, context deadline exceeded","id":"3B61DF7D34A8DA0A"}
  1. Both IAM role and IAM User:
{"level":"info","timestamp":"2021-11-26T07:13:25.530Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":" D3FFBA17E3C353A"}
{"level":"info","timestamp":"2021-11-26T07:13:25.534Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":"B04D437DA3F9FE98"}
{"level":"error","timestamp":"2021-11-26T07:13:27.969Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Post \"https://sts.us-east-1.amazonaws.com/\": Forbidden","id":"B04D437DA3F9FE98"}
{"level":"error","timestamp":"2021-11-26T07:13:29.713Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Post \"https://sts.us-east-1.amazonaws.com/\": Forbidden","id":" D3FFBA17E3C353A"}
  1. Added sts endpoint to NO_PROXY env var of filebeat:
{"level":"info","timestamp":"2021-11-26T07:15:40.914Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":" D3FFBA17E3C353A"}
{"level":"info","timestamp":"2021-11-26T07:15:40.920Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":"B04D437DA3F9FE98"}
{"level":"error","timestamp":"2021-11-26T07:17:11.695Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Post \"https://sts.us-east-1.amazonaws.com/\": dial tcp 54.239.16.72:443: i/o timeout","id":"B04D437DA3F9FE98"}
{"level":"error","timestamp":"2021-11-26T07:17:13.811Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Post \"https://sts.us-east-1.amazonaws.com/\": dial tcp 54.239.16.72:443: i/o timeout","id":" D3FFBA17E3C353A"}

It seems the documentation for 7.15.0 is bit off as it says to either use IAM user keys OR session token OR IAM role. However, just using IAM role is not working in this case.

17

@Ayush_Mathur

can you share your module config?
did you attach the role to the ec2 instance in case 1?

18

@Andrea_Spacca no, I didn't attach anything to my EC2 since it was nowhere mentioned in documentation: AWS module | Filebeat Reference [7.15] | Elastic

19

@Ayush_Mathur
if the EC2 instance has no attached the role defined in the module attached it will use the AssumeRole API how it is described by the documentation: AWS module | Filebeat Reference [7.15] | Elastic

Did you proceed with step 3 and 4?

20

Hello @Andrea_Spacca ,
I tried that AWS CLI command you mentioned above using the IAM user (which is also used by ES for snapshot storage in S3 bucket). The output is attached as image here.
image

The same IAM user, when used in filebeat is failing to poll s3 and give following error:

{"level":"error","timestamp":"2021-11-30T08:25:14.735Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.amazonaws.com/es-backup-xxxx?location=\": Forbidden","id":"EB4DF85955C0CAB1"}

where filebeat is running on same host as Elasticsearch node.