Could you please share your module config? (anonymised)
Could you also enable debug logs in Filebeat and post the whole output?
Did you set an endpoint in the config?
@Andrea_Spacca I can't really switch on the debug mode since beats is running on 50 nodes as daemon set. Is there any way to test the connection from pod terminal in verbose mode ?
The config is as follows:
- module: aws
cloudtrail:
enabled: false
var.bucket_arn: 'arn:aws:s3:::es-backup-xxxxx'
var.bucket_list_interval: 300s
var.number_of_workers: 5
var.access_key_id: ${ACCESS_KEY}
var.secret_access_key: ${SECRET_KEY}
var.visibility_timeout: 300s
var.api_timeout: 120s
var.endpoint: amazonaws.com
also mentioned initially when writing this post 
(just the bucket has changed)
please, remove var.endpoint. it is not really needed since you are using the default one and this is causing to build the host for the get bucket request as s3.us-east-1.amazonaws.com
I suppose the host machine where Filebeat runs is not in us-east-1, am I correct?
This setup from aws cli generates an initial body response like the following:
<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region \'eu-central-1\' is wrong; expecting \'us-east-1\'</Message><Region>us-east-1</Region>
aws cli handles the error and does a proper follow up request. My assumption is that the AWS SDK is not so smart.
@Ayush_Mathur
you should be able to run /bin/sh on the pod with kubectl exec
once you are logged in the pod you can test the two following aws cli command:
aws s3api get-bucket-location --debug --bucket es-backup-xxxxx --endpoint-url https://s3.us-east-1.amazonaws.com
and
aws s3api get-bucket-location --debug --bucket es-backup-xxxxx
@Andrea_Spacca removing the endpoint did't make any difference, the error is still present as Forbidden:
{"level":"error","timestamp":"2021-11-30T10:29:46.175Z","logger":"input.aws-s3","caller":"compat/compat.go:122","message":"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn: exceeded maximum number of attempts, 3, request send failed, Get \"https://s3.us-east-1.amazonaws.com/es-backup-xxxxxx?location=\": Forbidden","id":"563F408588050113"}
The host machine is in eu-west-1 region.
Also, filebeat doesn't know of aws command, is there some specific plugin I need to install in filebeat ?
Filebeat image is based on CentOS, you should be able to install aws cli with yum
I tried to reproduce Filebeat 7.15.0 enabling AWS SDK requests debugging and I see that the host set is indeed s3.us-east-1.amazonaws.com, and the GetBucketLocation is returning correct response from an host machine in eu-central-1
I would look in the permission of the role and bucket policy, in case there is any constraint on the region where the request comes from.
@Andrea_Spacca I tried installing aws-cli plugin into filebeat and tested if I was able to get the location of my bucket from filebeat container. The location was successfully retrieved since I guess I configured default region.
But again, even in my container spec, I have environment variable AWS_DEFAULT_REGION (as you mentioned earlier) set to my bucket region which should have overridden the default value of us-east-1 (Input 'aws-s3' failed with: failed to initialize s3 poller - #12 by Andrea_Spacca) - however, it didn't.
@Ayush_Mathur
AWS_DEFAULT_REGION env variable is not read by Filebeat.
Currently where is no possibility to set an AWS region for Filebeat S3 input.
This is taken care by setting it initially to us-east-1 in the S3 client used by Filebeat, that does then a first request to get the bucket region and then override the region of the S3 client accordingly.
This first request is indeed the one that fails for you.
Please try running the following aws cli command:
aws s3api get-bucket-location --debug --bucket es-backup-xxxxx --endpoint-url https://s3.us-east-1.amazonaws.com
this is what Filebeat does. Provide the IAM user or role that Filebeat uses.
1 Like
@Andrea_Spacca I tried running the above debug command and it looks like filebeat is trying to reach s3 API via proxy. The connection to proxy is being made over HTTP whereas the proxy is secured and HTTPS should be used instead. Is there any param that can be set for this ?
ProxyConnectionError: Failed to connect to proxy URL: "http://myproxy:3128"
Also, I tried adding the endpoint-url (s3.us-east-1.amazonaws.com) to NO_PROXY env variable. When I executed the command again, its getting:
Connect timeout on endpoint URL: "https://s3.us-east-1.amazonaws.com/es-backup-xxxxxx?location"
You run the aws cli command directly from the pod, did you?
The output you pasted is from aws cli, not from Filebeat, is it correct?
Please check the following AWS documentation: Using an HTTP proxy - AWS Command Line Interface
This seems more a problem on network setup on the cluster, and not a specific Filebeat issue. Are you running on EKS?
@Andrea_Spacca yes, I ran the command from pod terminal using aws-cli.
Also, I did following steps again:
- remove us-east-1 endpoint from NO_PROXY
- restarted filebeat pod
- configure aws-cli to use my IAM user
- setup HTTPS_PROXY as per AWS documentation
- ran the debug command again and below is the output:
SSLError: SSL validation failed for https://s3.us-east-1.amazonaws.com/es-backup-xxxxxx?location [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:618)
2021-12-02 09:57:46,925 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255
Can you try the following and check that a SSL handshake is happening?
curl -v https://s3.us-east-1.amazonaws.com/es-backup-xxxxxx?location
You can also try to pass --no-verify-ssl param to aws cli command
Do you use a NTLM proxy? aws cli doesn't support it
Hi @Andrea_Spacca, tried with curl command you mentioned and following was the output:
curl -v https://s3.us-east-1.amazonaws.com/es-backup-xxxxxx?location
* About to connect() to proxy myproxy port 3128 (#0)
* Trying aa.bb.cc.ddd...
* Connected to myproxy (aa.bb.cc.ddd) port 3128 (#0)
* Establish HTTP proxy tunnel to s3.us-east-1.amazonaws.com:443
* Proxy auth using Basic with user 'test'
> CONNECT s3.us-east-1.amazonaws.com:443 HTTP/1.1
> Host: s3.us-east-1.amazonaws.com:443
> Proxy-Authorization: Basic YWRtaW46TzdsbjBpUWNQQTdYbURNa2V2OVpxY0czd2Znam5uWVIzQW40b2VEYQ==
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 403 Forbidden
< Server: squid/3.5.12
< Mime-Version: 1.0
< Date: Fri, 03 Dec 2021 09:42:17 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3697
< X-Squid-Error: ERR_ACCESS_DENIED 0
< Vary: Accept-Language
< Content-Language: en
< X-Cache: MISS from nginx-1-79w9z
< X-Cache-Lookup: NONE from nginx-1-79w9z:3128
< Via: 1.1 nginx-1-79w9z (squid/3.5.12)
< Connection: keep-alive
<
* Received HTTP code 403 from proxy after CONNECT
* Connection #0 to host myproxy left intact
curl: (56) Received HTTP code 403 from proxy after CONNECT
The same output was present even with -k switch and the proxy being used is Nginx.
@Andrea_Spacca After further testing, it was found that our VPC doesn't allow any request coming in/ going out of eu-west-1 region due to security constraints of the project - hence the 403 Forbidden response.
However, I'm astonished to find out that AWS_DEFAULT_REGION (and other such properties for cloud providers) are not globally exposed. The same property can be defined in metricbeat to poll data from say, AWS RDS, but cannot be used in filebeat - which I think is a bug.
These type of properties should be made available in libbeat so they can be exposed in all beats which extends its functionality and users can avoid hitting this type of scenarios. The security constraints are rather rigid and cannot be tweaked which makes the configuration/ data ingestion impossible in such cases.
Furthermore, since platform security is way to go globally, the proxy configuration for AWS module doesn't provide any way to set SSL properties (like CA/CERT/KEY etc.) which again is an end game.
Can you please raise this with the development team as I'm pretty sure many users would like to have this behavior ?
I can't speak for @Andrea_Spacca but I don't see why the default region setting couldn't be added to Filebeat to enable such capability. As for the proxy ssl config, I believe that should be resolved with Move `fips_enabled` setting to AWS Common by legoguy1000 · Pull Request #28899 · elastic/beats · GitHub. U could try to build filebeat from source and test the config to see if it does what u need.
In the meantime u should be able to add the proxy root cert to the system trust store that filebeat is running on and it should trust it. Have you tried that?
@legoguy1000 at the moment SSL part is fine since AWS requests are being sent through from proxy without requiring any certificate (as both HTTP and HTTPS ports are enabled). I will reach out to the networking team to get the certificate and follow your suggestion (though they are skeptical in providing it due to security concerns).
The root CA cert should be considered public. Obviously not the key, there should be zero concerns to sharing the public key via a certificate.
1 Like
@Andrea_Spacca I tested a quick change to x-pack/libbeat/common/aws/credentials.go and it seems to work well to send the initial API calls the the default region variable if the region isn't already set. I can submit a PR for it?
func InitializeAWSConfig(config ConfigAWS) (awssdk.Config, error) {
AWSConfig, _ := GetAWSCredentials(config)
// Set default region if empty to make initial aws api call
if AWSConfig.Region == "" {
if config.DefaultRegion != "" {
AWSConfig.Region = config.DefaultRegion
} else {
AWSConfig.Region = "us-east-1"
}
}
....
}
I'm also pulling my hair on this black hole of an issue... i got this cross account setup like so:
# UserA (in account A)
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::bucket-in-ca-central-1/*",
"arn:aws:s3:::bucket-in-ca-central-1"
]
}
# BucketB (in account B)
{
"Sid": "AllowFilebeatPolling",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::1234567890:user/UserA"
},
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::${var.s3_bucket_name}",
"arn:aws:s3:::${var.s3_bucket_name}/*"
]
}
I can replicate all the actions with the equivalent cli as user A to bucket B, it works as expected, BUT whenever i tried to initialize filebeat it spits back 403...
... compat/compat.go: 122
"Input 'aws-s3' failed with: failed to initialize s3 poller: failed to get AWS region for bucket_arn:
So im wondering does it have something to do with "block all public access" on the bucket itself? I'll report back later with what i find...