Authentication in us-gov-west-1 with Metricbeat AWS module

[jstein_presidio (Joshua Stein)]

aws.yml is as follows

  regions: us-gov-west-1
  shared_credential_file: /mnt/home/metricbeat/.aws_creds
  aws_partition: aws-us-gov
  #endpoint: us-gov-west-1.amazonaws.com
  period: 5m
  metricsets:
    - ec2

2020-11-10T10:54:06.259-0500 WARN [aws.ec2] aws/aws.go:114 failed to get caller identity, please check permission setting: RequestError: send request failed caused by: Post "https://sts.us-gov-east-1.amazonaws.com/": dial tcp 52.46.100.6:443: connect: connection timed out

The timeout error is because this is running in us-gov-west-1 and doesn't have access to us-gov-east-1
If I uncomment the endpoint config and run with region hardcoded in there is a different error where it tries to look up iam creds against iam.us-gov-west-1.amazon.aws.com. The reason seams to be because iam is global and follows a different pattern with the url of us-gov.amazon.aws.com.

I think the issue here maybe that aws_partition defaults to us-gov-east-1 when set to aws-us-gov and has no way to be explicitly set, except by globally overriding with the endpoint parameter which appears break things in other ways.

[Kaiyan_Sheng]

Hello! Thanks for reporting this! This definitely looks like a bug to me. Could you please open a github issue in Beats for this?

Thank you!!

[jstein_presidio (Joshua Stein)]

Hey thanks for replying @Kaiyan_Sheng. I was able to resolve my issue. It turned out we were using a proxy to access aws services from within our network and the configuration above worked after including HTTP_PROXY and NO_PROXY environment variables to our metricbeat unit file.