Custom Cloudwatch Metrics

Hello,

Using v7.6.0

I'm attempting to collect some custom metrics using metricbeat, but not having any luck. I have tried pretty much every combination of the below possible

 - module: aws
  period: 60s
  access_key_id: '${AWS_ACCESS_KEY_ID}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY}'
  tags: "pre"
  metricsets:
    - cloudwatch
  metrics:            
    - namespace: ELK  
      name: ["ELKResponseTime"]
      statistic: ["Average"]          
      dimensions:
          - name: ELKResponseTime
            value: ELK    

I am able to successfully pull from cloudwatching using get-metric-data using the same info.

{
        "MetricDataQueries": [
    {
        "Id": "m1",
        "MetricStat": {
                "Metric": {
                        "Namespace": "ELK",
                        "MetricName": "ELKResponseTime",
                        "Dimensions" : [{
                                "Name": "ELKResponseTime",
                                "Value": "ELK"
                                }]
                },
                "Period": 60,
                "Stat": "Average"
                        }
                }
    ],
    "StartTime": "2020-02-27T14:00:0000",
    "EndTime": "2020-02-27T14:15:0000"
}

response

{
    "Messages": [],
    "MetricDataResults": [
        {
            "Id": "m1",
            "Timestamps": [
                "2020-02-27T14:14:00Z",
                "2020-02-27T14:13:00Z",
                "2020-02-27T14:12:00Z",
                "2020-02-27T14:11:00Z",
                "2020-02-27T14:10:00Z",
                "2020-02-27T14:09:00Z",
                "2020-02-27T14:08:00Z",
                "2020-02-27T14:07:00Z",
                "2020-02-27T14:06:00Z",
                "2020-02-27T14:05:00Z",
                "2020-02-27T14:04:00Z",
                "2020-02-27T14:03:00Z",
                "2020-02-27T14:02:00Z",
                "2020-02-27T14:01:00Z",
                "2020-02-27T14:00:00Z"
            ],
            "StatusCode": "Complete",
            "Label": "ELKResponseTime",
            "Values": [
                0.09296989440917969,
                0.0331873893737793,
                0.11049914360046387,
                0.03397560119628906,
                0.045053720474243164,
                0.04449105262756348,
                0.0390627384185791,
                0.03135561943054199,
                0.0417938232421875,
                0.04479169845581055,
                0.12064003944396973,
                0.11304378509521484,
                0.03579378128051758,
                0.026007890701293945,
                0.19768404960632324
            ]
        }
    ]
}

Hi!

Not sure if this is supported :thinking:. Maybe @Kaiyan_Sheng could verify.

C.

@AddChickpeas Thanks for posting here! I'm gonna try reproduce it today to see what happens with custom metrics. Cloudwatch metricset is using ListMetrics and GetMetricData two API calls. Good to know that get-metric-data API is working. Do you know if ListMetrics works with namespace=ELK?

Hi @AddChickpeas, I just created some custom metrics and stored in CloudWatch namespace called Custom. With metricbeat aws module enabled with aws.yml:

- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  regions:
    - us-east-1
  metricsets:
    - cloudwatch
  metrics:
    - namespace: Custom

I was able to collect these metrics from Custom namespace:

{
  "_index": "metricbeat-8.0.0-2020.02.28-000001",
  "_type": "_doc",
  "_id": "yVibjHABNNjmvAqrcSsh",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-02-28T16:22:39.031Z",
    "host": {
      "architecture": "x86_64",
      "os": {
        "name": "Mac OS X",
        "kernel": "17.7.0",
        "build": "17G10021",
        "platform": "darwin",
        "version": "10.13.6",
        "family": "darwin"
      },
      "id": "9C7FAB7B-29D1-5926-8E84-158A9CA3E25D",
      "name": "KaiyanMacBookPro",
      "ip": [
        "fe80::18da:9553:5e03:fd2d",
        "192.168.0.5",
        "fe80::6c96:28ff:fe6d:fcdc",
        "fe80::1b53:9dee:f305:e2c7",
        "fe80::aede:48ff:fe00:1122"
      ],
      "mac": [
        "f2:18:98:75:78:55",
        "f0:18:98:75:78:55",
        "02:18:98:75:78:55",
        "6e:96:28:6d:fc:dc",
        "c6:00:e4:40:f6:04",
        "c6:00:e4:40:f6:01",
        "c6:00:e4:40:f6:00",
        "c6:00:e4:40:f6:05",
        "c6:00:e4:40:f6:01",
        "ac:de:48:00:11:22"
      ],
      "hostname": "KaiyanMacBookPro"
    },
    "agent": {
      "type": "metricbeat",
      "ephemeral_id": "5f9d5e70-b0e3-4c0b-a7cf-0fbef5fc92cb",
      "hostname": "KaiyanMacBookPro",
      "id": "23fd6ee9-2255-43f3-9565-78a051337c04",
      "version": "8.0.0"
    },
    "aws": {
      "custom": {
        "metrics": {
          "memory-usage": {
            "max": 83.05,
            "min": 82.94,
            "sum": 414.81,
            "count": 5,
            "avg": 82.962
          },
          "Tcp_connections": {
            "avg": 56,
            "max": 56,
            "min": 56,
            "sum": 280,
            "count": 5
          },
          "IO_WAIT": {
            "count": 5,
            "avg": 0.01,
            "max": 0.01,
            "min": 0.01,
            "sum": 0.05
          },
          "TCP_connection_on_port_80": {
            "avg": 3,
            "max": 3,
            "min": 3,
            "sum": 15,
            "count": 5
          },
          "No_of_users": {
            "avg": 1,
            "max": 1,
            "min": 1,
            "sum": 5,
            "count": 5
          }
        }
      },
      "cloudwatch": {
        "namespace": "Custom"
      },
      "dimensions": {
        "Instance": "i-05381e495aa0f95ef"
      }
    },
    "event": {
      "dataset": "aws.cloudwatch",
      "module": "aws",
      "duration": 651057530
    },
    "metricset": {
      "period": 300000,
      "name": "cloudwatch"
    },
    "service": {
      "type": "aws"
    },
    "cloud": {
      "provider": "aws",
      "region": "us-east-1",
      "account": {
        "name": "elastic-beats",
        "id": "428152502467"
      }
    },
    "ecs": {
      "version": "1.4.0"
    }
  },
  "fields": {
    "@timestamp": [
      "2020-02-28T16:22:39.031Z"
    ]
  },
  "sort": [
    1582906959031
  ]
}

Do you see any error message in Metricbeat log? Maybe you can try with a simpler config first, like:

 - module: aws
  period: 60s
  access_key_id: '${AWS_ACCESS_KEY_ID}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY}'
  metricsets:
    - cloudwatch
  metrics:            
    - namespace: ELK 

Hi Kaiyan,

Thanks for getting back to me.

I tested list-metrics and it is returning logs. Looks like the person that added them hasn't really cleaned it up all that much, but the metrics are there. There are a bunch of duplicates so not sure if that could be effecting things. I'll see if they added any others I can test with.

{
"Metrics": [       
    {
        "Dimensions": [
            {
                "Value": "ELK",
                "Name": "ELKResponseTime"
            }
        ],
        "Namespace": "ELK",
        "MetricName": "ELKResponseTime"
    },
    {
        "Dimensions": [
            {
                "Value": "ELK",
                "Name": "ELKResponseError"
            }
        ],
        "Namespace": "ELK",
        "MetricName": "ELKResponseError"
    },
    {
        "Dimensions": [
            {
                "Value": "ELK",
                "Name": "ELKResponseError"
            }
        ],
        "Namespace": "ELK",
        "MetricName": "ELKResponseTime"
    },
    {
        "Dimensions": [
            {
                "Value": "ELK",
                "Name": "ELKResponseTime"
            }
        ],
        "Namespace": "ELK",
        "MetricName": "ELKResponseError"
    }
]
}

I updated the config just to include this,. but no luck

metricbeat.modules:
- module: aws
  period: 60s
  access_key_id: '${AWS_ACCESS_KEY_ID_DATADEV}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY_DATADEV}'
  tags: ["dev"]        
  metricsets:
    - cloudwatch
  metrics:
    - namespace: ELK 

Nothing in logs either even with debug on:

2020-02-28T20:12:55.515Z    INFO    [monitoring    log/log.go:118    Starting metrics logging every 30s
2020-02-28T20:12:55.515Z    INFO    instance/beat.go:439    metricbeat start running.
2020-02-28T20:12:55.516Z    DEBUG    [module    module/wrapper.go:120    Starting Wrapper[name=aws, len(metricSetWrappers)=1]
2020-02-28T20:12:55.516Z    DEBUG    [module    module/wrapper.go:174    aws/cloudwatch will start after 7.222403846s
2020-02-28T20:13:02.738Z    DEBUG    [module    module/wrapper.go:182    Starting metricSetWrapper[module=aws, name=cloudwatch, host=]
2020-02-28T20:13:25.526Z    INFO    [monitoring    log/log.go:145    Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":100,"time":{"ms":109}},"total":{"ticks":290,"time":{"ms":304},"value":290},"user":{"ticks":190,"time":{"ms":195}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":7},"info":{"ephemeral_id":"8c72a0ab-8741-4cd4-97d3-2405f89f5365","uptime":{"ms":36306}},"memstats":{"gc_next":14784800,"memory_alloc":11824680,"memory_total":29226752,"rss":75718656},"runtime":{"goroutines":22}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0}}},"system":{"cpu":{"cores":4},"load":{"1":11.46,"15":13.99,"5":13.32,"norm":{"1":2.865,"15":3.4975,"5":3.33}}}}}}

Cool, then this cloudwatch metricset should work. Could you test it with a simpler config like this one please?

 - module: aws
  period: 60s
  access_key_id: '${AWS_ACCESS_KEY_ID}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY}'
  metricsets:
    - cloudwatch
  metrics:            
    - namespace: ELK 

Also, if it failed, could you post some logs please. That would be helpful too! Thanks!

Please see my edit in the previous post. I edited to add that information because I submitted it early by accident.

I tested with that config, but no luck.

1 Like

Thanks! Could you remove tags: ["dev"] and rerun Metricbeat please? I just saw this in the config hmmm
Are you trying to use tags to filter?

tags as filter for cloudwatch metricset is poorly documented (not documented :frowning_face: ). I will push a PR to add this for sure. Sorry about that! Here is the PR for adding tags as filter and the PR description has some example on how to use it: https://github.com/elastic/beats/pull/13758

I was just adding tags into the event like you can with other beats. The _doc ends up having a tags field with pre as the value.

metricset.name cloudwatch
metricset.period 60,000
service.type aws
tags pre

I'll give it a try without and get back to you. We're making some changes so I'll have to wait until it actually starts outputting the custom metrics again.

I will give that tags as filter a try! I tried it at one point, but didn't have much luck. I ended up just listing items I wanted to pull metrics on manually.

Ahh sorry I misunderstood. The PR I linked is to use tags as filters for collecting metrics from CloudWatch. If you want to collect all metrics from ELK namespace, please ignore my comment about filtering with tags :grimacing: Thanks!

So I think the issue was with the configuration of the custom metrics themselves. I switched to using a different custom metric and it is working as expected. Not really sure what the difference between them is.

Thanks for your help!

1 Like

Interesting! Good to know! Thanks for all the info!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.