This is similar to Metricbeat AWS module endpoint not working correctly. I tested on 7.13.1, but I think this should persist to current master branch.
So I tried 3 different configurations and am unable to get the desired metricbeat stats for AWS. First, my aws.yml contains endpoint, region, and role_arn (endpoint and region are examples - not exactly what is used):
- module: aws
period: 300s
endpoint: amazonaws.com.cn
regions: cn-north-1
metricsets:
- ec2
role_arn: "arn:aws:iam::ACCOUNT_NUM:role/instance_role"
I would expect it to call sts.cn-north-1.amazonaws.com.cn for verification, but instead, it still calls sts.amazonaws.com, which wouldn't exist. Looking into the libbeat/common/aws/credentials.go code, I then removed the role_arn and tried the following aws.yml:
- module: aws
period: 300s
endpoint: amazonaws.com.cn
regions: cn-north-1
metricsets:
- ec2
This way, it should be able to authenticate using the instance profile (without specifying the role ARN). It's able to get pass the STS check from libbeat/common/aws/credentials.go:111, but then future services end up duplicating the region (e.g. it calls 'ec2.cn-north-1.cn-north-1.amazonaws.com.cn' instead of 'ec2.cn-north-1.amazonaws.com.cn'). To get around this, I omitted the region and instead prefixed it onto the endpoint field:
- module: aws
period: 300s
endpoint: cn-north-1.amazonaws.com.cn
metricsets:
- ec2
However, when it tries the STS verification, it now complains about a mismatched region (us-east-1 vs cn-north-1).
I think part of the issue is at x-pack/libbeat/common/aws/credentials.go:98 where the region is set initially to 'us-east-1'. When the role arn is empty, it then attempts to use STS with us-east-1, which then fails because the region doesnt exist for me.