Lost aws module (rds metricset) metrics after upgrade from 7.17 -> 8.8.2

[m_standfuss (Michael Standfuss)]

After upgrading from 7.17 to 8.8.2 we are no longer getting any of our rds metrics from the aws module.

We are running on EKS in AWS, nothing changed in terms of the modules configured, overall metricbeats configuration, service account its using in k8s etc, except the version.

aws module code

    - module: aws
      period: 60s
      metricsets:
        - rds

I did enabled trace level logging on 8.8, and it appears that it cannot find any of my rds instances in us-west-1 that it was finding prior to the version upgrade (I did manually downgrade it as well to confirm that it is getting pulled in from the existing version still).

Example of metric gathered after downgrading version with no other changes (can send entire payload if its useful)

[m_standfuss (Michael Standfuss)]

Here is the ServiceAccount configuration

➜  ~ kubectl describe serviceaccount metricbeat --context prod --namespace kube-system
Name:                metricbeat
Namespace:           kube-system
Labels:              k8s-app=metricbeat
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::<account id removed>:role/terraform-20220317204940606500000001
Image pull secrets:  <none>
Mountable secrets:   metricbeat-token-sqvjs
Tokens:              metricbeat-token-sqvjs
Events:              <none>

it did not change between version upgrades, I can post the permissions attached to that role if its needed, but they are the ones outlined in the RDS module AWS rds metricset | Metricbeat Reference [8.10] | Elastic

[m_standfuss (Michael Standfuss)]

There are instances in us-west-1 that it is still scraping just fine, it only seems to be the us-east-1 region that it is no longer finding instances in (which is where our production instances are so thats really the most important region to scrape metrics for us)

[m_standfuss (Michael Standfuss)]

I tried stepping the downgrades to see where in the version stack it starts to break. When I went down to 8.7 (8.7.2) it now works again.

{"log.level":"debug","@timestamp":"2023-09-21T17:48:45.311Z","log.logger":"cloudwatch","log.origin":{"file.name":"cloudwatch/cloudwatch.go","file.line":520},"message":"In region us-east-1, service db:production tags match tags_filter","service.name":"metricbeat","ecs.version":"1.6.0"}