# Input 'aws-s3' failed with: failed to initialize s3 poller

**URL:** <https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 1, 2021, 11:32am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130 "2021-11-01T11:32:33Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![Andrea\_Spacca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_spacca/32/76976_2.png) [@Andrea\_Spacca](https://discuss.elastic.co/u/Andrea_Spacca)\
**Post date:** [November 30, 2021, 9:31am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/21 "2021-11-30T09:31:30Z")

</div>

@Ayush_Mathur

Could you please share your module config? (anonymised)  
Could you also enable debug logs in Filebeat and post the whole output?

Did you set an `endpoint` in the config?

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [November 30, 2021, 9:44am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/22 "2021-11-30T09:44:02Z")

</div>

@Andrea_Spacca I can't really switch on the debug mode since beats is running on 50 nodes as daemon set. Is there any way to test the connection from pod terminal in verbose mode ?  
The config is as follows:

```auto
  - module: aws
    cloudtrail:
      enabled: false
      var.bucket_arn: 'arn:aws:s3:::es-backup-xxxxx'
      var.bucket_list_interval: 300s
      var.number_of_workers: 5
      var.access_key_id: ${ACCESS_KEY}
      var.secret_access_key: ${SECRET_KEY}
      var.visibility_timeout: 300s
      var.api_timeout: 120s
      var.endpoint: amazonaws.com

```

also mentioned initially when writing this post 🙂 (just the bucket has changed)

---

<div class="post-metadata">

**Author:** ![Andrea\_Spacca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_spacca/32/76976_2.png) [@Andrea\_Spacca](https://discuss.elastic.co/u/Andrea_Spacca)\
**Post date:** [November 30, 2021, 10:10am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/23 "2021-11-30T10:10:47Z")

</div>

@Ayush_Mathur

please, remove `var.endpoint`. it is not really needed since you are using the default one and this is causing to build the host for the get bucket request as `s3.us-east-1.amazonaws.com`

I suppose the host machine where Filebeat runs is not in `us-east-1`, am I correct?  
This setup from aws cli generates an initial body response like the following:  
`<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region \'eu-central-1\' is wrong; expecting \'us-east-1\'</Message><Region>us-east-1</Region>`

aws cli handles the error and does a proper follow up request. My assumption is that the AWS SDK is not so smart.

---

<div class="post-metadata">

**Author:** ![Andrea\_Spacca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_spacca/32/76976_2.png) [@Andrea\_Spacca](https://discuss.elastic.co/u/Andrea_Spacca)\
**Post date:** [November 30, 2021, 10:31am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/24 "2021-11-30T10:31:31Z")

</div>

@Ayush_Mathur  
you should be able to run `/bin/sh` on the pod with `kubectl exec`  
once you are logged in the pod you can test the two following aws cli command:

```auto
aws s3api get-bucket-location --debug --bucket es-backup-xxxxx --endpoint-url https://s3.us-east-1.amazonaws.com

```

and

```auto
aws s3api get-bucket-location --debug --bucket es-backup-xxxxx

```

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [November 30, 2021, 10:33am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/25 "2021-11-30T10:33:10Z")

</div>

@Andrea_Spacca removing the endpoint did't make any difference, the error is still present as Forbidden:

```auto
{"level":"error","timestamp":"2021-11-30T10:29:46.175Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.amazonaws.com/es-backup-xxxxxx?location=\": Forbidden","id":"563F408588050113"}

```

The host machine is in eu-west-1 region.  
Also, filebeat doesn't know of aws command, is there some specific plugin I need to install in filebeat ?

---

<div class="post-metadata">

**Author:** ![Andrea\_Spacca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_spacca/32/76976_2.png) [@Andrea\_Spacca](https://discuss.elastic.co/u/Andrea_Spacca)\
**Post date:** [November 30, 2021, 11:04am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/26 "2021-11-30T11:04:35Z")

</div>

@Ayush_Mathur

Filebeat image is based on CentOS, you should be able to install aws cli with yum

I tried to reproduce Filebeat 7.15.0 enabling AWS SDK requests debugging and I see that the host set is indeed ` s3.us-east-1.amazonaws.com`, and the `GetBucketLocation` is returning correct response from an host machine in `eu-central-1`

I would look in the permission of the role and bucket policy, in case there is any constraint on the region where the request comes from.

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [December 1, 2021, 12:17pm UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/27 "2021-12-01T12:17:51Z")

</div>

@Andrea_Spacca I tried installing aws-cli plugin into filebeat and tested if I was able to get the location of my bucket from filebeat container. The location was successfully retrieved since I guess I configured default region.  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/f/4f2bd37baea408772992c3cd1c33a196bf863042.png)

But again, even in my container spec, I have environment variable AWS\_DEFAULT\_REGION (as you mentioned earlier) set to my bucket region which should have overridden the default value of us-east-1 ([Input 'aws-s3' failed with: failed to initialize s3 poller - #12 by Andrea\_Spacca](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/12)) - however, it didn't.

---

<div class="post-metadata">

**Author:** ![Andrea\_Spacca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_spacca/32/76976_2.png) [@Andrea\_Spacca](https://discuss.elastic.co/u/Andrea_Spacca)\
**Post date:** [December 1, 2021, 1:05pm UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/28 "2021-12-01T13:05:51Z")

</div>

@Ayush_Mathur  
`AWS_DEFAULT_REGION` env variable is not read by Filebeat.  
Currently where is no possibility to set an AWS region for Filebeat S3 input.  
This is taken care by setting it initially to `us-east-1` in the S3 client used by Filebeat, that does then a first request to get the bucket region and then override the region of the S3 client accordingly.  
This first request is indeed the one that fails for you.

Please try running the following aws cli command:  
`aws s3api get-bucket-location --debug --bucket es-backup-xxxxx --endpoint-url https://s3.us-east-1.amazonaws.com`

this is what Filebeat does. Provide the IAM user or role that Filebeat uses.

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [December 2, 2021, 9:22am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/29 "2021-12-02T09:22:51Z")

</div>

@Andrea_Spacca I tried running the above debug command and it looks like filebeat is trying to reach s3 API via proxy. The connection to proxy is being made over HTTP whereas the proxy is secured and HTTPS should be used instead. Is there any param that can be set for this ?

```auto
ProxyConnectionError: Failed to connect to proxy URL: "http://myproxy:3128"

```

Also, I tried adding the endpoint-url ([s3.us-east-1.amazonaws.com](http://s3.us-east-1.amazonaws.com)) to NO\_PROXY env variable. When I executed the command again, its getting:

```auto
Connect timeout on endpoint URL: "https://s3.us-east-1.amazonaws.com/es-backup-xxxxxx?location"

```

---

<div class="post-metadata">

**Author:** ![Andrea\_Spacca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_spacca/32/76976_2.png) [@Andrea\_Spacca](https://discuss.elastic.co/u/Andrea_Spacca)\
**Post date:** [December 2, 2021, 9:42am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/30 "2021-12-02T09:42:58Z")

</div>

@Ayush_Mathur

You run the aws cli command directly from the pod, did you?  
The output you pasted is from aws cli, not from Filebeat, is it correct?

Please check the following AWS documentation: [Using an HTTP proxy - AWS Command Line Interface](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-proxy.html)

This seems more a problem on network setup on the cluster, and not a specific Filebeat issue. Are you running on EKS?

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [December 2, 2021, 9:59am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/31 "2021-12-02T09:59:47Z")

</div>

@Andrea_Spacca yes, I ran the command from pod terminal using aws-cli.  
Also, I did following steps again:

1. remove us-east-1 endpoint from NO\_PROXY
2. restarted filebeat pod
3. configure aws-cli to use my IAM user
4. setup HTTPS\_PROXY as per AWS documentation
5. ran the debug command again and below is the output:

```auto
SSLError: SSL validation failed for https://s3.us-east-1.amazonaws.com/es-backup-xxxxxx?location [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:618)
2021-12-02 09:57:46,925 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

```

---

<div class="post-metadata">

**Author:** ![Andrea\_Spacca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_spacca/32/76976_2.png) [@Andrea\_Spacca](https://discuss.elastic.co/u/Andrea_Spacca)\
**Post date:** [December 2, 2021, 10:52am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/32 "2021-12-02T10:52:53Z")

</div>

@Ayush_Mathur

Can you try the following and check that a SSL handshake is happening?  
`curl -v https://s3.us-east-1.amazonaws.com/es-backup-xxxxxx?location`

You can also try to pass `--no-verify-ssl` param to aws cli command

Do you use a NTLM proxy? aws cli doesn't support it

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [December 3, 2021, 9:51am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/33 "2021-12-03T09:51:52Z")

</div>

Hi @Andrea_Spacca, tried with curl command you mentioned and following was the output:

```auto
curl -v https://s3.us-east-1.amazonaws.com/es-backup-xxxxxx?location
* About to connect() to proxy myproxy port 3128 (#0)
* Trying aa.bb.cc.ddd...
* Connected to myproxy (aa.bb.cc.ddd) port 3128 (#0)
* Establish HTTP proxy tunnel to s3.us-east-1.amazonaws.com:443
* Proxy auth using Basic with user 'test'
> CONNECT s3.us-east-1.amazonaws.com:443 HTTP/1.1
> Host: s3.us-east-1.amazonaws.com:443
> Proxy-Authorization: Basic YWRtaW46TzdsbjBpUWNQQTdYbURNa2V2OVpxY0czd2Znam5uWVIzQW40b2VEYQ==
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 403 Forbidden
< Server: squid/3.5.12
< Mime-Version: 1.0
< Date: Fri, 03 Dec 2021 09:42:17 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3697
< X-Squid-Error: ERR_ACCESS_DENIED 0
< Vary: Accept-Language
< Content-Language: en
< X-Cache: MISS from nginx-1-79w9z
< X-Cache-Lookup: NONE from nginx-1-79w9z:3128
< Via: 1.1 nginx-1-79w9z (squid/3.5.12)
< Connection: keep-alive
< 
* Received HTTP code 403 from proxy after CONNECT
* Connection #0 to host myproxy left intact
curl: (56) Received HTTP code 403 from proxy after CONNECT

```

The same output was present even with _-k_ switch and the proxy being used is Nginx.

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [December 7, 2021, 10:40am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/34 "2021-12-07T10:40:52Z")

</div>

@Andrea_Spacca After further testing, it was found that our VPC doesn't allow any request coming in/ going out of eu-west-1 region due to security constraints of the project - hence the 403 Forbidden response.

However, I'm astonished to find out that AWS\_DEFAULT\_REGION (and other such properties for cloud providers) are not globally exposed. The same property can be defined in metricbeat to poll data from say, AWS RDS, but cannot be used in filebeat - which I think is a bug.

These type of properties should be made available in libbeat so they can be exposed in all beats which extends its functionality and users can avoid hitting this type of scenarios. The security constraints are rather rigid and cannot be tweaked which makes the configuration/ data ingestion impossible in such cases.

Furthermore, since platform security is way to go globally, the proxy configuration for AWS module doesn't provide any way to set SSL properties (like CA/CERT/KEY etc.) which again is an end game.

Can you please raise this with the development team as I'm pretty sure many users would like to have this behavior ?

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [December 7, 2021, 11:46am UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/35 "2021-12-07T11:46:58Z")

</div>

I can't speak for @Andrea_Spacca but I don't see why the default region setting couldn't be added to Filebeat to enable such capability. As for the proxy ssl config, I believe that should be resolved with [Move `fips_enabled` setting to AWS Common by legoguy1000 · Pull Request #28899 · elastic/beats · GitHub](https://github.com/elastic/beats/pull/28899/files#diff-87bf43e29ebf053e042e3327ebcf103820d00d47a4265f883f26be23e8c7cbf2R43). U could try to build filebeat from source and test the config to see if it does what u need.

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [December 7, 2021, 12:59pm UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/36 "2021-12-07T12:59:44Z")

</div>

In the meantime u should be able to add the proxy root cert to the system trust store that filebeat is running on and it should trust it. Have you tried that?

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [December 7, 2021, 1:36pm UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/37 "2021-12-07T13:36:38Z")

</div>

@legoguy1000 at the moment SSL part is fine since AWS requests are being sent through from proxy without requiring any certificate (as both HTTP and HTTPS ports are enabled). I will reach out to the networking team to get the certificate and follow your suggestion (though they are skeptical in providing it due to security concerns).

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [December 7, 2021, 1:51pm UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/38 "2021-12-07T13:51:13Z")

</div>

The root CA cert should be considered public. Obviously not the key, there should be zero concerns to sharing the public key via a certificate.

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [December 7, 2021, 9:57pm UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/39 "2021-12-07T21:57:08Z")

</div>

@Andrea_Spacca I tested a quick change to `x-pack/libbeat/common/aws/credentials.go` and it seems to work well to send the initial API calls the the default region variable if the region isn't already set. I can submit a PR for it?

```auto
func InitializeAWSConfig(config ConfigAWS) (awssdk.Config, error) {
	AWSConfig, _ := GetAWSCredentials(config)

	// Set default region if empty to make initial aws api call
	if AWSConfig.Region == "" {
		if config.DefaultRegion != "" {
			AWSConfig.Region = config.DefaultRegion
		} else {
			AWSConfig.Region = "us-east-1"
		}
	}
....
}

```

---

<div class="post-metadata">

**Author:** ![MrAtheist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mratheist/32/98751_2.png) [@MrAtheist](https://discuss.elastic.co/u/MrAtheist)\
**Post date:** [December 11, 2021, 8:08pm UTC](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130/40 "2021-12-11T20:08:19Z")

</div>

I'm also pulling my hair on this black hole of an issue... i got this cross account setup like so:

```auto
# UserA (in account A)

{
       "Effect": "Allow",
       "Action": [
           "s3:GetObject",
           "s3:ListBucket",
           "s3:GetBucketLocation"
       ],
       "Resource": [
           "arn:aws:s3:::bucket-in-ca-central-1/*",
           "arn:aws:s3:::bucket-in-ca-central-1"
       ]
   }

```

```auto
# BucketB (in account B)

{
    "Sid": "AllowFilebeatPolling",
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::1234567890:user/UserA"
    },
    "Action": [
      "s3:GetObject",
      "s3:ListBucket",
      "s3:GetBucketLocation"
    ],
    "Resource": [
      "arn:aws:s3:::${var.s3_bucket_name}",
      "arn:aws:s3:::${var.s3_bucket_name}/*"
    ]
}

```

I can replicate all the actions with the equivalent cli as user A to bucket B, it works as expected, BUT whenever i tried to initialize filebeat it spits back 403...

```auto
... compat/compat.go: 122
"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn:

```

So im wondering does it have something to do with "block all public access" on the bucket itself? I'll report back later with what i find...

[Previous page](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130.md?page=1)

[Next page](https://discuss.elastic.co/t/input-aws-s3-failed-with-failed-to-initialize-s3-poller/288130.md?page=3)