How to get only the information we want from the metricsbeat cloudwatch set?

[Ryan_Waldron]

We are trying to get only a few specific metrics from cloudwatch about our RDS instances (CPU, Memory, Etc). I have followed the documentation to try and come up with a configuration for metricbeats that works, and nothing does. We either get metricbeats vaccuming down a million metrics that we have no interest in, increasing the costs to us from both AWS and elastic for nothing, or we get a single metric from a single DB instance and the rest of the instances are not monitored. Also setting the "period" option to try and throttle the frequency of the metrics has absolutely no affect on anything, the system just pulls when it wants.

I have tried using seperate aws modules per DB instance:

- module: aws
  period: 30s
  metricsets:
    - cloudwatch
  access_key_id: 'xxxxx'
  secret_access_key: 'xxxxxxxx'
  metrics:
    - namespace: AWS/RDS
      name: ["CPUUtilization"]
      dimensions:
        - name: DBInstanceIdentifier
          value: db1
      statistic: ["Average", "Maximum"]

- module: aws
  period: 30s
  metricsets:
    - cloudwatch
  access_key_id: 'xxxxxx'
  secret_access_key: 'xxxxxx'
  metrics:
    - namespace: AWS/RDS
      name: ["CPUUtilization"]
      dimensions:
        - name: DBInstanceIdentifier
          value: db2
      statistic: ["Average", "Maximum"]

- module: aws
  period: 30s
  metricsets:
    - cloudwatch
  access_key_id: 'xxxxxxx'
  secret_access_key: 'xxxxxxxx'
  metrics:
    - namespace: AWS/RDS
      name: ["CPUUtilization"]
      dimensions:
        - name: DBInstanceIdentifier
          value: db3
      statistic: ["Average", "Maximum"]

I have tried using them all as seperate metrics to a single module definition:

- module: aws
  period: 30s
  metricsets:
    - cloudwatch
  access_key_id: 'xxxxx'
  secret_access_key: 'xxxxxxxx'
  metrics:
    - namespace: AWS/RDS
      name: ["CPUUtilization"]
      dimensions:
        - name: DBInstanceIdentifier
          value: db1
      statistic: ["Average", "Maximum"]
    - namespace: AWS/RDS
      name: ["CPUUtilization"]
      dimensions:
        - name: DBInstanceIdentifier
          value: db2
      statistic: ["Average", "Maximum"]
    - namespace: AWS/RDS
      name: ["CPUUtilization"]
      dimensions:
        - name: DBInstanceIdentifier
          value: db3
      statistic: ["Average", "Maximum"]

Both of these configurations will only return the single metric from the first database (db1) , nothing about the other dbs at all. The only other option is to just let it vaccum up everything from the RDS namespace, and pay to pull and store a bunch of useless data to the the one peice of data that we want for each database.

How can we configure metricbeats to get the specific cloudwatch metrics we want (All of them)?
Why do you include the "period" setting when the system apparently completely ignores that and checks at the frequency that it feels like at the time?
When can you update the documentation to save future users the time and expense of trying to figure all of this out?

[Ryan_Waldron]

You can see that dispite the "period" value being set to 30 seconds, it is checking the metric every minute. I have tried other values larger and smaller, and it seems that a metric a minute is all we get no matter what is in the configuration.

[Kaiyan_Sheng]

Hello @Ryan_Waldron, sorry that you have these troubles using cloudwatch metricset. What version of Metricbeat are you using? I will try to reproduce this problem and get back to you asap.

[Ryan_Waldron]

V 7.4.0

[Kaiyan_Sheng]

I tried with Metricbeat 7.4.2 downloaded from https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-installation.html with aws.yml looks like:

- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  credential_profile_name: test-mb
  metrics:
    - namespace: AWS/RDS
      name: ["CPUUtilization", "FreeableMemory"]
      dimensions:
        - name: DBInstanceIdentifier
          value: database-1
      statistic: ["Average", "Maximum"]
    - namespace: AWS/RDS
      name: ["CPUUtilization", "FreeableMemory"]
      dimensions:
        - name: DBInstanceIdentifier
          value: test3
      statistic: ["Average", "Maximum"]

I got only metrics from RDS DBInstance database-1 and test3:

Also with period=300s setting, metrics were collected every 5min. Hmm could you give this latest version of 7.4 a try? I remember there were some bugs fixed and backported. Thank you for all the info!

[Kaiyan_Sheng]

Sorry but it does seem to have a bug even in 7.4.2 when I specify more than one metric name in the config. The workaround for that will be separating the config:

- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  credential_profile_name: test-mb
  metrics:
    - namespace: AWS/RDS
      name: ["CPUUtilization"]
      dimensions:
        - name: DBInstanceIdentifier
          value: database-1
      statistic: ["Average", "Maximum"]
    - namespace: AWS/RDS
      name: ["FreeableMemory"]
      dimensions:
        - name: DBInstanceIdentifier
          value: database-1
      statistic: ["Average", "Maximum"]
    - namespace: AWS/RDS
      name: ["CPUUtilization"]
      dimensions:
        - name: DBInstanceIdentifier
          value: test3
      statistic: ["Average", "Maximum"]
    - namespace: AWS/RDS
      name: ["FreeableMemory"]
      dimensions:
        - name: DBInstanceIdentifier
          value: test3
      statistic: ["Average", "Maximum"]

[Kaiyan_Sheng]

https://github.com/elastic/beats/issues/14376 working on fixing this!