Metricbeat w/ aws module says failed to list account aliases and failed to get caller identity

[syost (Shane Yost)]

I've enabled the metricbeat module called aws. When starting metricbeat I receive the following errors...

2020-05-27T23:59:07.253Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-05-27T23:59:07.837Z	WARN	[aws.cloudwatch]	aws/aws.go:100	failed to list account aliases, please check permission setting: InvalidClientTokenId: The security token included in the request is invalid.
	status code: 403, request id: d381b7db-fb4d-452c-9898-aa3e3b915ba8
2020-05-27T23:59:08.242Z	WARN	[aws.cloudwatch]	aws/aws.go:116	failed to get caller identity, please check permission setting: InvalidClientTokenId: The security token included in the request is invalid.
	status code: 403, request id: f452aeb1-7004-432f-af98-359e7e6b4e79
2020-05-27T23:59:08.242Z	INFO	cfgfile/reload.go:175	Config reloader started
2020-05-27T23:59:11.732Z	WARN	[aws.cloudwatch]	aws/aws.go:100	failed to list account aliases, please check permission setting: InvalidClientTokenId: The security token included in the request is invalid.
	status code: 403, request id: ba5d0055-cc05-4f74-a056-bc4ad8fd9d6f
2020-05-27T23:59:12.066Z	WARN	[aws.cloudwatch]	aws/aws.go:116	failed to get caller identity, please check permission setting: InvalidClientTokenId: The security token included in the request is invalid.
	status code: 403, request id: 4965c8e0-df48-4ea6-a4dc-88e7644d133e
2020-05-27T23:59:12.066Z	INFO	cfgfile/reload.go:235	Loading of config files completed.

I have set environment variables in my .bashrc with the correct access keys as such...

export AWS_ACCESS_KEY_ID=BLABLABLABLA
export AWS_SECRET_ACCESS_KEY=blablablabla
export AWS_DEFAULT_REGION=us-gov-west-1

and my module config aws.yml looks like...

- module: aws                                                                    
  period: 5m                                                                     
  access_key_id: '${AWS_ACCESS_KEY_ID:""}'                                                                  
  secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'                               
  regions:                                                                       
    - us-gov-west-1                                                              
  metricsets:                                                                    
    - cloudwatch                                                                 
  metrics:                                                                       
    - namespace: my-log-group

Is this a IAM roles issue? Is namespace the name of the log group in cloudwatch? Really confused as to why its refusing to get account aliases and indentity. I'm able to execute the following aws cli commands just fine...

[~/] > aws iam list-account-aliases
{
    "AccountAliases": []
}

[~/] > aws sts get-caller-identity
{
    "UserId": "AIDARNasdfasfdafdafd",
    "Account": "123456789",
    "Arn": "arn:aws-us-gov:iam::123456789:user/shane.yost"
}

[Kaiyan_Sheng]

Hi! Namespace is not the log group in cloudwatch, it is the service you want to collect metrics for, such as AWS/EC2, AWS/ELB...

Based on the error, it does seem to be related to the permission setting. For example, if you want to collect ec2 metrics from CloudWatch, these are the permissions needed: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-aws-ec2.html#_aws_permissions_4

[syost (Shane Yost)]

@Kaiyan_Sheng
Thanks Kaiyan ~ In cloudwatch all my log groups have the title /aws/lambda/loggroup1 or /aws/step/loggroup1. Therefore, could I specify a namespace of /aws/step and will that track all log groups underneath that namespace? Is that how that works? Apologies for my confusion, I thought metricbeat was capable of watching specific log groups.

[Kaiyan_Sheng]

Hmmm I don't believe cloudwatch metricset in Metricbeat tracks log groups. It's designed to collect monitoring metrics from AWS CloudWatch similar to GetMetricData API: https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html

Maybe monitoring cloudwatch logs is closer to what you are looking for? If that's the case, you can try config below:

- module: aws                                                                    
  period: 5m                                                                     
  access_key_id: '${AWS_ACCESS_KEY_ID:""}'                                                                  
  secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'                                                                                         
  metricsets:                                                                    
    - cloudwatch                                                                 
  metrics:                                                                       
    - namespace: AWS/Logs