Aws cloudwatch metricset is sampling data -- can it import data exactly?

Elastic Stack Beats
2

Hello @cmr.paul.wilkinson, thanks for posting your question here! I think the main issue here is the period. Based on the MetricDataResults you showed, these metrics are reported every 2 minutes to Cloudwatch. But for cloudwatch metricset using Metricbeat, the period is set to 1-minute and that causes the duplicate events.

I see that you are trying to pull data from several different namespaces, do you want to collect them with the same period? If not, you can set different periods for different namespaces. For example:

- module: aws
  period: 60s
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/DynamoDB
  regions:
    - eu-west-2
- module: aws
  period: 120s
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/Events
    - namespace: AWS/Lambda
  regions:
    - eu-west-2
- module: aws
  period: 5m
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/Usage
  regions:
    - eu-west-2

You can check cloudwatch metrics on AWS portal (documentation as well) to see what's the collection period should be.

For timestamp, unfortunately it will not be a complete match right now. We are using the time we collect this event as the timestamp instead of from cloudwatch timestamp. This is a great enhancement issue though. I will create a ticket for that! Thanks!

1 Like