# Input 'aws-s3' failed with: failed to initialize s3 poller

**URL:** <https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 1, 2021, 11:32am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130 "2021-11-01T11:32:33Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [November 1, 2021, 11:32am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/1 "2021-11-01T11:32:33Z")

</div>

Hi,

I've been trying to setup AWS module on 7.15 Elastic Stack cluster running as containers to ingest cloudtrail, cloudwatch, elb, s3access and vpcflow events (configured only for cloudtrail atm).  
However, there seems to be some bug in filebeat wherein when **bucket.arn** is configured, it always appends us-east-1 region by default making the S3 bucket inaccessible.

The relevant filebeat config is as follows:

```auto
  - module: aws
    cloudtrail:
      enabled: true
      var.bucket_arn: 'arn:aws:s3:::test-bucket'
      var.bucket_list_interval: 300s
      var.number_of_workers: 5
      var.access_key_id: ${ACCESS_KEY}
      var.secret_access_key: ${SECRET_KEY}
      var.visibility_timeout: 300s
      var.api_timeout: 120s
      var.endpoint: amazonaws.com
      var.default_region: "${AWS_REGION:eu-west-1}"

```

with S3 type input set as:

```auto
- type: aws-s3
  bucket_arn: arn:aws:s3:::test-bucket
  number_of_workers: 5
  bucket_list_interval: 300s
  expand_event_list_from_field: Records
  access_key_id: ${ACCESS_KEY}
  secret_access_key: ${SECRET_KEY}
  endpoint: amazonaws.com
  default_region: "${AWS_REGION:eu-west-1}"
  parsers:
    - multiline:
        pattern: "^<Event"
        negate: true
        match: after

```

However, no matter what config I provide (when using bucket\_arn), us-east-1 is always appended to the connection string. Sample logs are as follows:

```auto
{"level":"error","timestamp":"2021-10-29T09:32:18.382Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.eu-west-1.amazonaws.com/test-bucket?location=\": Forbidden","id":"7C946250A2CBE8D6"}

{"level":"error","timestamp":"2021-10-29T09:42:04.606Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.s3.eu-west-1.amazonaws.com/test-bucket?location=\": x509: certificate is valid for s3-eu-west-1.amazonaws.com, *.s3-eu-west-1.amazonaws.com, s3.eu-west-1.amazonaws.com, *.s3.eu-west-1.amazonaws.com, s3.dualstack.eu-west-1.amazonaws.com, *.s3.dualstack.eu-west-1.amazonaws.com, *.s3.amazonaws.com, *.s3-control.eu-west-1.amazonaws.com, s3-control.eu-west-1.amazonaws.com, *.s3-control.dualstack.eu-west-1.amazonaws.com, s3-control.dualstack.eu-west-1.amazonaws.com, *.s3-accesspoint.eu-west-1.amazonaws.com, *.s3-accesspoint.dualstack.eu-west-1.amazonaws.com, *.s3.eu-west-1.vpce.amazonaws.com, not s3.us-east-1.s3.eu-west-1.amazonaws.com","id":"25BD02C5CFA1AFB3"}

{"level":"error","timestamp":"2021-10-29T09:49:33.194Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.https://s3.eu-west-1.amazonaws.com/test-bucket/test-bucket?location=\": Forbidden","id":"73D457479F8D5CDB"}

{"level":"error","timestamp":"2021-10-29T10:00:02.343Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.amazonaws.com/test-bucket?location=\": Forbidden","id":"BDCC88C015F87747"}

```

@Kaiyan_Sheng can you please help me to resolve this, may be I'm missing something ?

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [November 2, 2021, 2:55am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/2 "2021-11-02T02:55:12Z")

</div>

I don't see anywhere in the code where `default_region` config is a thing. Have u tried not setting it? You shouldn't need to, it will determine the region where the S3 bucket exists. How did u generate those errors, what were the configs for each?

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [November 2, 2021, 9:16am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/3 "2021-11-02T09:16:16Z")

</div>

@legoguy1000 I added `default_region` to try if that makes filebeat to use my region instead of default `us-east-1` (took it from metricbeat configuration where it works fine).  
As far as those errors are concerned, they are:

1. without specifying `default_region`
2. specifying `default_region`
3. adding full s3 bucket endpoint with https (duplicate https in url)
4. adding full s3 bucket endpoint without https (duplicate s3.\*\*\* in url)

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [November 2, 2021, 9:21am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/4 "2021-11-02T09:21:39Z")

</div>

Also, I don't see any documentation regarding SSL params to be used if my proxy requires them. Any idea what parameter should be used in this case ?

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [November 2, 2021, 10:07am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/5 "2021-11-02T10:07:50Z")

</div>

So if u specify a fully url as the endpoint it will not work in 7.15. As for `default_region`, that exists for metricbeat but has been deprecated and removed in 8.0.0 but I couldn't find anything for Filebeat. I'd remove it and try again. The aws-s3 input will query the us-east-1 endpoint to determine where the S3 bucket is and then when it actually polls it, will use the appropriate region endpoint for where the bucket is located. Also u don't need to specify the `endpoint` config if it's `amazonaws.com`. As for the proxy config, there was a bug that @kaiyan-sheng found, [Change proxy\_url from url.URL to string by kaiyan-sheng · Pull Request #28725 · elastic/beats · GitHub](https://github.com/elastic/beats/pull/28725), so it may not work. But currently there are no SSL configs for the `aws-3` proxy url.

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [November 2, 2021, 4:09pm UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/6 "2021-11-02T16:09:57Z")

</div>

@legoguy1000 I tried removing the `default_region` as per your suggestion but kept `endpoint` config. The result is still the same:

```auto
{"level":"info","timestamp":"2021-11-02T16:04:11.250Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":"2000C88E6AA7EA04"}
{"level":"info","timestamp":"2021-11-02T16:04:11.253Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":" BE301CC83DBCE5"}
{"level":"error","timestamp":"2021-11-02T16:04:12.169Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.amazonaws.com/test-bucket-1-stos?location=\": Forbidden","id":" BE301CC83DBCE5"}
{"level":"error","timestamp":"2021-11-02T16:04:13.377Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.amazonaws.com/test-bucket-1-stos?location=\": Forbidden","id":"2000C88E6AA7EA04"}

```

So I can infer two things here:

1. I may need `proxy_url` to be configured (but it needs SSL which is not yet available)
2. I may need to switch to SQS queue instead which will require whole new level of TA analysis since it's not yet in approved list of services in our project.

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [November 2, 2021, 4:35pm UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/7 "2021-11-02T16:35:32Z")

</div>

> [@Ayush\_Mathur](#):
>
> `failed to get AWS region for bucket_arn`

Its failing to find the region for ur bucket. Thats where its erroring. Are you sure the credentials are correct and have the right IAM permissions?

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [November 2, 2021, 5:18pm UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/8 "2021-11-02T17:18:28Z")

</div>

@legoguy1000 yes, the credentials are correct and have permissions since the same IAM user is being used by ES for snapshot in the same bucket.

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [November 2, 2021, 9:53pm UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/9 "2021-11-02T21:53:34Z")

</div>

Just to confirm, it has at min the below? Ref [AWS S3 input | Filebeat Reference [7.15] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html#_aws_permissions_2)

```auto
s3:GetObject
s3:ListBucket
s3:GetBucketLocation

```

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [November 3, 2021, 7:14am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/10 "2021-11-03T07:14:15Z")

</div>

@legoguy1000 yes, all the permissions specified are allowed for the IAM user.

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [November 3, 2021, 12:11pm UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/11 "2021-11-03T12:11:55Z")

</div>

> [@Ayush\_Mathur](#):
>
> yes, all the permissions specified are allowed for the IAM user.

OK.

> [@Ayush\_Mathur](#):
>
> `https://s3.us-east-1.amazonaws.com/test-bucket-1-stos?location=`

The only other thing thats weird is that AWS doesn't support path style buckets so this should be like`https://test-bucket-1-stos.s3.us-east-1.amazonaws.com/?location`

Or possibly per the docs `To use this implementation of the operation, you must be the bucket owner`. Are the credentials you're using associated with the account that owns the bucket? At this point i'm running out of ideas.

---

<div class="post-metadata">

**Author:** ![Andrea\_Spacca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_spacca/32/76976_2.png) [@Andrea\_Spacca](https://discuss.elastic.co/u/Andrea_Spacca)\
**Post date:** [November 5, 2021, 11:39am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/12 "2021-11-05T11:39:10Z")

</div>

@Ayush_Mathur

can you try running the following aws-cli command using the IAM resource used by filebeat?

```auto
aws s3api get-bucket-location --bucket test-bucket-1-stos

```

As for the config params:  
`endpoint` is not needed if your bucket must not be accessed from a custom domain, like your case  
`default_region` is not supported in filebeat. You can set the env variable `AWS_DEFAULT_REGION` to override it if you want.

If not region is set (either missing or empty `AWS_DEFAULT_REGION` env or setting in `~/.aws/config` or in EC2 instance metadata) `us-east-1` is used as fallback.

This will be the region used for the `GetBucketLocation` request, once we get the region of the bucket we will use it in the following AWS call to that bucket.

`GetBucketLocation` is not region bound: you can perform the request from any region to a bucket in any other regions, and receive a proper result.

---

<div class="post-metadata">

**Author:** ![Andrea\_Spacca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_spacca/32/76976_2.png) [@Andrea\_Spacca](https://discuss.elastic.co/u/Andrea_Spacca)\
**Post date:** [November 5, 2021, 11:49am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/13 "2021-11-05T11:49:31Z")

</div>

> [@legoguy1000](#):
>
> The only other thing thats weird is that AWS doesn't support path style buckets so this should be like `https://test-bucket-1-stos.s3.us-east-1.amazonaws.com/?location`

The call is on the s3 api endpoint, not on the bucket endpoint, that's why the url called is `https://s3.us-east-1.amazonaws.com/test-bucket-1-stos?location=`

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [November 5, 2021, 12:05pm UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/14 "2021-11-05T12:05:50Z")

</div>

Ok, I was going off of [GetBucketLocation - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html) which shows virtual host style.

---

<div class="post-metadata">

**Author:** ![Andrea\_Spacca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_spacca/32/76976_2.png) [@Andrea\_Spacca](https://discuss.elastic.co/u/Andrea_Spacca)\
**Post date:** [November 5, 2021, 12:18pm UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/15 "2021-11-05T12:18:00Z")

</div>

@legoguy1000  
Apparently you can call with any combination of `bucket` being in the host or the path and it will work anyway. If you have different buckets between host and path, the result will be for the host one

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [November 26, 2021, 7:26am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/16 "2021-11-26T07:26:21Z")

</div>

I tried configuring the module all over again but tried using IAM role which was used to create the bucket. Following are the 3 different erroneous responses I received:

1. Only IAM role in input and module:

```auto
{"level":"info","timestamp":"2021-11-26T07:10:41.506Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":"861367A1E96D11A8"}
{"level":"info","timestamp":"2021-11-26T07:10:41.507Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":"3B61DF7D34A8DA0A"}
{"level":"error","timestamp":"2021-11-26T07:10:51.509Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: EC2RoleRequestError: no EC2 instance role found\ncaused by: request canceled, context deadline exceeded","id":"861367A1E96D11A8"}
{"level":"error","timestamp":"2021-11-26T07:10:51.509Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: EC2RoleRequestError: no EC2 instance role found\ncaused by: request canceled, context deadline exceeded","id":"3B61DF7D34A8DA0A"}

```

1. Both IAM role and IAM User:

```auto
{"level":"info","timestamp":"2021-11-26T07:13:25.530Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":" D3FFBA17E3C353A"}
{"level":"info","timestamp":"2021-11-26T07:13:25.534Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":"B04D437DA3F9FE98"}
{"level":"error","timestamp":"2021-11-26T07:13:27.969Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Post \"https://sts.us-east-1.amazonaws.com/\": Forbidden","id":"B04D437DA3F9FE98"}
{"level":"error","timestamp":"2021-11-26T07:13:29.713Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Post \"https://sts.us-east-1.amazonaws.com/\": Forbidden","id":" D3FFBA17E3C353A"}

```

1. Added sts endpoint to NO\_PROXY env var of filebeat:

```auto
{"level":"info","timestamp":"2021-11-26T07:15:40.914Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":" D3FFBA17E3C353A"}
{"level":"info","timestamp":"2021-11-26T07:15:40.920Z","logger":"input.aws-s3","caller":"compat/compat.go:111","message":"Input aws-s3 starting","id":"B04D437DA3F9FE98"}
{"level":"error","timestamp":"2021-11-26T07:17:11.695Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Post \"https://sts.us-east-1.amazonaws.com/\": dial tcp 54.239.16.72:443: i/o timeout","id":"B04D437DA3F9FE98"}
{"level":"error","timestamp":"2021-11-26T07:17:13.811Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Post \"https://sts.us-east-1.amazonaws.com/\": dial tcp 54.239.16.72:443: i/o timeout","id":" D3FFBA17E3C353A"}

```

It seems the documentation for 7.15.0 is bit off as it says to either use IAM user keys OR session token OR IAM role. However, just using IAM role is not working in this case.

---

<div class="post-metadata">

**Author:** ![Andrea\_Spacca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_spacca/32/76976_2.png) [@Andrea\_Spacca](https://discuss.elastic.co/u/Andrea_Spacca)\
**Post date:** [November 26, 2021, 9:15am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/17 "2021-11-26T09:15:49Z")

</div>

@Ayush_Mathur

can you share your module config?  
did you attach the role to the ec2 instance in case 1?

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [November 26, 2021, 9:18am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/18 "2021-11-26T09:18:51Z")

</div>

@Andrea_Spacca no, I didn't attach anything to my EC2 since it was nowhere mentioned in documentation: [AWS module | Filebeat Reference [7.15] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-aws.html)

---

<div class="post-metadata">

**Author:** ![Andrea\_Spacca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_spacca/32/76976_2.png) [@Andrea\_Spacca](https://discuss.elastic.co/u/Andrea_Spacca)\
**Post date:** [November 26, 2021, 9:41am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/19 "2021-11-26T09:41:39Z")

</div>

@Ayush_Mathur  
if the EC2 instance has no attached the role defined in the module attached it will use the `AssumeRole` API how it is described by the documentation: [AWS module | Filebeat Reference [7.15] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-aws.html#_aws_credentials_types_2)

Did you proceed with step 3 and 4?

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [November 30, 2021, 8:26am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/20 "2021-11-30T08:26:14Z")

</div>

Hello @Andrea_Spacca ,  
I tried that AWS CLI command you mentioned above using the IAM user (which is also used by ES for snapshot storage in S3 bucket). The output is attached as image here.  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/2/b2c6b27e12f1e74eac3cf2c021b225f8ac00e82b.png)

The same IAM user, when used in filebeat is failing to poll s3 and give following error:

```auto
{"level":"error","timestamp":"2021-11-30T08:25:14.735Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.amazonaws.com/es-backup-xxxx?location=\": Forbidden","id":"EB4DF85955C0CAB1"}

```

where filebeat is running on same host as Elasticsearch node.

[Next page](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130.md?page=2)