Metric beat unable to detect all metricset from cloudwatch

Elastic Stack Beats
1

I want to detect AWS/SES metric set from cloud watch or basically monitor SES metrics via elastic. I tried to configure "*" namespace but still, it doesn't query all of the namespaces.

I am have tested with 7.5.1 and 7.6.2 as well.

2

Hi @Abhishek_Tanwar, thanks for reporting your issue here. Could you share your config file here please? Thanks! I will try to reproduce it on my side too.

3

Could you also verify around the timestamp of you have Metricbeat running, there are metrics in CloudWatch under AWS/SES namespace please? Thanks!

4

Screenshot 2020-04-11 at 11.06.56
Screenshot 2020-04-11 at 11.06.56760×394 27.7 KB

5

I assume you are using the shared credential file ~/.aws/credentials default profile name for authentication then? If that's the case, the config looks right!

While running Metricbeat, do you see any metrics under AWS/SES namespace inside CloudWatch portal?

6

Yes I can see metrics in cloud watch. There are 4 metrics as standard in AWS/SES.

Regards,

Abhishek Tanwar

7

Hmm I still can't reproduce it. Here is what I see in CloudWatch:

Screen Shot 2020-04-13 at 1.17.43 PM
Screen Shot 2020-04-13 at 1.17.43 PM4476×1476 287 KB

and when I start Metricbeat with aws.yml config:

- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/SES

I got an event:

{
  "_index": "metricbeat-8.0.0-2020.04.13-000001",
  "_type": "_doc",
  "_id": "dtr4dHEBnhw1ZtvVMsHy",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-04-13T19:15:42.328Z",
    "event": {
      "dataset": "aws.cloudwatch",
      "module": "aws",
      "duration": 11090806378
    },
    "metricset": {
      "name": "cloudwatch",
      "period": 300000
    },
    "service": {
      "type": "aws"
    },
    "cloud": {
      "provider": "aws",
      "region": "us-east-1",
      "account": {
        "name": "elastic-beats",
        "id": "428152502467"
      }
    },
    "aws": {
      "ses": {
        "metrics": {
          "Send": {
            "avg": 1,
            "max": 1,
            "min": 1,
            "sum": 1,
            "count": 1
          },
          "Delivery": {
            "count": 1,
            "avg": 1,
            "max": 1,
            "min": 1,
            "sum": 1
          }
        }
      },
      "cloudwatch": {
        "namespace": "AWS/SES"
      }
    }
  },
  "fields": {
    "@timestamp": [
      "2020-04-13T19:15:42.328Z"
    ]
  },
  "sort": [
    1586805342328
  ]
}
8

Based on the timestamp, there is a delay. Is that might be the issue for you? Or are you seeing any error message from Metricbeat log?