Custom Cloudwatch Metrics

AddChickpeas · February 27, 2020, 8:29pm

Hello,

Using v7.6.0

I'm attempting to collect some custom metrics using metricbeat, but not having any luck. I have tried pretty much every combination of the below possible

 - module: aws
  period: 60s
  access_key_id: '${AWS_ACCESS_KEY_ID}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY}'
  tags: "pre"
  metricsets:
    - cloudwatch
  metrics:            
    - namespace: ELK  
      name: ["ELKResponseTime"]
      statistic: ["Average"]          
      dimensions:
          - name: ELKResponseTime
            value: ELK

I am able to successfully pull from cloudwatching using get-metric-data using the same info.

{
        "MetricDataQueries": [
    {
        "Id": "m1",
        "MetricStat": {
                "Metric": {
                        "Namespace": "ELK",
                        "MetricName": "ELKResponseTime",
                        "Dimensions" : [{
                                "Name": "ELKResponseTime",
                                "Value": "ELK"
                                }]
                },
                "Period": 60,
                "Stat": "Average"
                        }
                }
    ],
    "StartTime": "2020-02-27T14:00:0000",
    "EndTime": "2020-02-27T14:15:0000"
}

response

{
    "Messages": [],
    "MetricDataResults": [
        {
            "Id": "m1",
            "Timestamps": [
                "2020-02-27T14:14:00Z",
                "2020-02-27T14:13:00Z",
                "2020-02-27T14:12:00Z",
                "2020-02-27T14:11:00Z",
                "2020-02-27T14:10:00Z",
                "2020-02-27T14:09:00Z",
                "2020-02-27T14:08:00Z",
                "2020-02-27T14:07:00Z",
                "2020-02-27T14:06:00Z",
                "2020-02-27T14:05:00Z",
                "2020-02-27T14:04:00Z",
                "2020-02-27T14:03:00Z",
                "2020-02-27T14:02:00Z",
                "2020-02-27T14:01:00Z",
                "2020-02-27T14:00:00Z"
            ],
            "StatusCode": "Complete",
            "Label": "ELKResponseTime",
            "Values": [
                0.09296989440917969,
                0.0331873893737793,
                0.11049914360046387,
                0.03397560119628906,
                0.045053720474243164,
                0.04449105262756348,
                0.0390627384185791,
                0.03135561943054199,
                0.0417938232421875,
                0.04479169845581055,
                0.12064003944396973,
                0.11304378509521484,
                0.03579378128051758,
                0.026007890701293945,
                0.19768404960632324
            ]
        }
    ]
}

ChrsMark · February 28, 2020, 9:25am

Hi!

Not sure if this is supported . Maybe @Kaiyan_Sheng could verify.

C.

Kaiyan_Sheng · February 28, 2020, 2:44pm

@AddChickpeas Thanks for posting here! I'm gonna try reproduce it today to see what happens with custom metrics. Cloudwatch metricset is using ListMetrics and GetMetricData two API calls. Good to know that get-metric-data API is working. Do you know if ListMetrics works with namespace=ELK?

Kaiyan_Sheng · February 28, 2020, 4:32pm

Hi @AddChickpeas, I just created some custom metrics and stored in CloudWatch namespace called Custom. With metricbeat aws module enabled with aws.yml:

- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  regions:
    - us-east-1
  metricsets:
    - cloudwatch
  metrics:
    - namespace: Custom

I was able to collect these metrics from Custom namespace:

{
  "_index": "metricbeat-8.0.0-2020.02.28-000001",
  "_type": "_doc",
  "_id": "yVibjHABNNjmvAqrcSsh",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-02-28T16:22:39.031Z",
    "host": {
      "architecture": "x86_64",
      "os": {
        "name": "Mac OS X",
        "kernel": "17.7.0",
        "build": "17G10021",
        "platform": "darwin",
        "version": "10.13.6",
        "family": "darwin"
      },
      "id": "9C7FAB7B-29D1-5926-8E84-158A9CA3E25D",
      "name": "KaiyanMacBookPro",
      "ip": [
        "fe80::18da:9553:5e03:fd2d",
        "192.168.0.5",
        "fe80::6c96:28ff:fe6d:fcdc",
        "fe80::1b53:9dee:f305:e2c7",
        "fe80::aede:48ff:fe00:1122"
      ],
      "mac": [
        "f2:18:98:75:78:55",
        "f0:18:98:75:78:55",
        "02:18:98:75:78:55",
        "6e:96:28:6d:fc:dc",
        "c6:00:e4:40:f6:04",
        "c6:00:e4:40:f6:01",
        "c6:00:e4:40:f6:00",
        "c6:00:e4:40:f6:05",
        "c6:00:e4:40:f6:01",
        "ac:de:48:00:11:22"
      ],
      "hostname": "KaiyanMacBookPro"
    },
    "agent": {
      "type": "metricbeat",
      "ephemeral_id": "5f9d5e70-b0e3-4c0b-a7cf-0fbef5fc92cb",
      "hostname": "KaiyanMacBookPro",
      "id": "23fd6ee9-2255-43f3-9565-78a051337c04",
      "version": "8.0.0"
    },
    "aws": {
      "custom": {
        "metrics": {
          "memory-usage": {
            "max": 83.05,
            "min": 82.94,
            "sum": 414.81,
            "count": 5,
            "avg": 82.962
          },
          "Tcp_connections": {
            "avg": 56,
            "max": 56,
            "min": 56,
            "sum": 280,
            "count": 5
          },
          "IO_WAIT": {
            "count": 5,
            "avg": 0.01,
            "max": 0.01,
            "min": 0.01,
            "sum": 0.05
          },
          "TCP_connection_on_port_80": {
            "avg": 3,
            "max": 3,
            "min": 3,
            "sum": 15,
            "count": 5
          },
          "No_of_users": {
            "avg": 1,
            "max": 1,
            "min": 1,
            "sum": 5,
            "count": 5
          }
        }
      },
      "cloudwatch": {
        "namespace": "Custom"
      },
      "dimensions": {
        "Instance": "i-05381e495aa0f95ef"
      }
    },
    "event": {
      "dataset": "aws.cloudwatch",
      "module": "aws",
      "duration": 651057530
    },
    "metricset": {
      "period": 300000,
      "name": "cloudwatch"
    },
    "service": {
      "type": "aws"
    },
    "cloud": {
      "provider": "aws",
      "region": "us-east-1",
      "account": {
        "name": "elastic-beats",
        "id": "428152502467"
      }
    },
    "ecs": {
      "version": "1.4.0"
    }
  },
  "fields": {
    "@timestamp": [
      "2020-02-28T16:22:39.031Z"
    ]
  },
  "sort": [
    1582906959031
  ]
}

Do you see any error message in Metricbeat log? Maybe you can try with a simpler config first, like:

 - module: aws
  period: 60s
  access_key_id: '${AWS_ACCESS_KEY_ID}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY}'
  metricsets:
    - cloudwatch
  metrics:            
    - namespace: ELK

AddChickpeas · February 28, 2020, 8:16pm

Hi Kaiyan,

Thanks for getting back to me.

I tested list-metrics and it is returning logs. Looks like the person that added them hasn't really cleaned it up all that much, but the metrics are there. There are a bunch of duplicates so not sure if that could be effecting things. I'll see if they added any others I can test with.

{
"Metrics": [       
    {
        "Dimensions": [
            {
                "Value": "ELK",
                "Name": "ELKResponseTime"
            }
        ],
        "Namespace": "ELK",
        "MetricName": "ELKResponseTime"
    },
    {
        "Dimensions": [
            {
                "Value": "ELK",
                "Name": "ELKResponseError"
            }
        ],
        "Namespace": "ELK",
        "MetricName": "ELKResponseError"
    },
    {
        "Dimensions": [
            {
                "Value": "ELK",
                "Name": "ELKResponseError"
            }
        ],
        "Namespace": "ELK",
        "MetricName": "ELKResponseTime"
    },
    {
        "Dimensions": [
            {
                "Value": "ELK",
                "Name": "ELKResponseTime"
            }
        ],
        "Namespace": "ELK",
        "MetricName": "ELKResponseError"
    }
]
}

I updated the config just to include this,. but no luck

metricbeat.modules:
- module: aws
  period: 60s
  access_key_id: '${AWS_ACCESS_KEY_ID_DATADEV}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY_DATADEV}'
  tags: ["dev"]        
  metricsets:
    - cloudwatch
  metrics:
    - namespace: ELK

Nothing in logs either even with debug on:

2020-02-28T20:12:55.515Z    INFO    [monitoring    log/log.go:118    Starting metrics logging every 30s
2020-02-28T20:12:55.515Z    INFO    instance/beat.go:439    metricbeat start running.
2020-02-28T20:12:55.516Z    DEBUG    [module    module/wrapper.go:120    Starting Wrapper[name=aws, len(metricSetWrappers)=1]
2020-02-28T20:12:55.516Z    DEBUG    [module    module/wrapper.go:174    aws/cloudwatch will start after 7.222403846s
2020-02-28T20:13:02.738Z    DEBUG    [module    module/wrapper.go:182    Starting metricSetWrapper[module=aws, name=cloudwatch, host=]
2020-02-28T20:13:25.526Z    INFO    [monitoring    log/log.go:145    Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":100,"time":{"ms":109}},"total":{"ticks":290,"time":{"ms":304},"value":290},"user":{"ticks":190,"time":{"ms":195}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":7},"info":{"ephemeral_id":"8c72a0ab-8741-4cd4-97d3-2405f89f5365","uptime":{"ms":36306}},"memstats":{"gc_next":14784800,"memory_alloc":11824680,"memory_total":29226752,"rss":75718656},"runtime":{"goroutines":22}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0}}},"system":{"cpu":{"cores":4},"load":{"1":11.46,"15":13.99,"5":13.32,"norm":{"1":2.865,"15":3.4975,"5":3.33}}}}}}

Kaiyan_Sheng · February 28, 2020, 8:20pm

Cool, then this cloudwatch metricset should work. Could you test it with a simpler config like this one please?

 - module: aws
  period: 60s
  access_key_id: '${AWS_ACCESS_KEY_ID}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY}'
  metricsets:
    - cloudwatch
  metrics:            
    - namespace: ELK

Also, if it failed, could you post some logs please. That would be helpful too! Thanks!

AddChickpeas · February 28, 2020, 8:22pm

Please see my edit in the previous post. I edited to add that information because I submitted it early by accident.

I tested with that config, but no luck.

Kaiyan_Sheng · February 28, 2020, 8:27pm

Thanks! Could you remove tags: ["dev"] and rerun Metricbeat please? I just saw this in the config hmmm
Are you trying to use tags to filter?

Kaiyan_Sheng · February 28, 2020, 8:29pm

tags as filter for cloudwatch metricset is poorly documented (not documented ). I will push a PR to add this for sure. Sorry about that! Here is the PR for adding tags as filter and the PR description has some example on how to use it: https://github.com/elastic/beats/pull/13758

AddChickpeas · February 28, 2020, 8:47pm

I was just adding tags into the event like you can with other beats. The _doc ends up having a tags field with pre as the value.

metricset.name cloudwatch
metricset.period 60,000
service.type aws
tags pre

I'll give it a try without and get back to you. We're making some changes so I'll have to wait until it actually starts outputting the custom metrics again.

I will give that tags as filter a try! I tried it at one point, but didn't have much luck. I ended up just listing items I wanted to pull metrics on manually.

Kaiyan_Sheng · February 28, 2020, 8:51pm

Ahh sorry I misunderstood. The PR I linked is to use tags as filters for collecting metrics from CloudWatch. If you want to collect all metrics from ELK namespace, please ignore my comment about filtering with tags Thanks!

AddChickpeas · February 28, 2020, 9:15pm

So I think the issue was with the configuration of the custom metrics themselves. I switched to using a different custom metric and it is working as expected. Not really sure what the difference between them is.

Thanks for your help!

Kaiyan_Sheng · February 28, 2020, 9:55pm

Interesting! Good to know! Thanks for all the info!

system · March 27, 2020, 9:55pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.