Metricbeat AWS RDS Module not retrieving db_instance.identifiers

[achalamalasetti (Anil Chalamalasetti)]

we are using metricbeat (7.10.0) -> AWS module, RDS metrics. The metric data that is pulled looks incomplete. it doesn't retrieve aws.rds.db_instance.engine_name for some and for some it doesn't retrieve aws.rds.db_instance.identifier, or aws.rds.db_instance.identifier, or aws.rds.db_instance.class (see sample screenshot below). What am I missing here?

text- module: aws
  period: 1m
  metricsets:
    - rds
  regions:
    - us-west-2

[achalamalasetti (Anil Chalamalasetti)]

Debugging for another day with out replies, looks like I am able narrow down that problem to not having the permission - rds:ListTagsForResource.

https://github.com/elastic/beats/blob/master/x-pack/metricbeat/module/aws/rds/rds.go Line : 192-198, It was an most likely an error in our case, though the return statement doesn't make sense. If its a warning, throw a warning, continue and if its an error, throw any error. Returning partially constructed data is incorrect, most likely resulting in the weird behavior i noticed.

reqListTags := svc.ListTagsForResourceRequest(&listTagsInput)
		outputListTags, err := reqListTags.Send(context.TODO())
		if err != nil {
			m.Logger().Warn("ListTagsForResourceRequest failed, rds:ListTagsForResource permission is required for getting tags.")
			dbDetailsMap[*dbInstance.DBInstanceIdentifier] = dbDetails
			return dbInstanceIDs, dbDetailsMap, nil
		}

Another unrelated bug, same class, Line 180-182, looks like an error.

if dbInstance.DBInstanceClass != nil {
			dbDetails.dbStatus = *dbInstance.DBInstanceStatus
		}

The aws.rds.db_instance.engine_name is another mystery still unsolved.

[Kaiyan_Sheng]

Hello! Thanks for reporting this problem. We consider these fields optional, that's why we logged a warning in the log and still published the metrics. For engine_name, this is actually a dimension from the original RDS CloudWatch metrics. Some of the CloudWatch metrics have EngineName as dimension, some have other fields as dimension. For example: DBInstanceIdentifier, DBClusterIdentifier and etc. Please see the section Amazon RDS Dimensions in https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MonitoringOverview.html#USER_Monitoring for more details. Thanks!